Breaking the “Engineering is Always Late” Narrative: Mastering Predictable Delivery | GLOW 2024

Hey y'all, we're, uh, we're gonna wrap up the, the day strong. Uh, so I'm, I'm Alexa, I'm a product manager here at Jellyfish, and I'm here alongside Andrew Korbel, VP of R&D Operations at Clary. And today we'll be discussing the importance of delivery predictability as well as tactical steps for how to measure it within your organizations through learning direct through Andrew's experience and about creating a culture of delivery predictability measurement at Clary. So leading up to the session, I've really enjoyed learning from Andrew. He's got an amazing story to tell. I'm so excited for y'all to hear it. So I know you're gonna love it. Uh, we'll have Q&A at the end, so we're gonna try to leave some time for that. So feel free to drop in your questions that come up during the session. and we'll do our best to answer, uh, a live reply and chat at the end, or, uh, if we run out of time, we'll follow up. Um, but yeah, so excited to be here to kick things off. Andrew, why don't you start off by telling the audience a bit about yourself?

Yeah, so as Alexis said, I'm the, uh, my name's Andrew. I'm the VP of R&D Ops at, uh, Clary. I've been in sort of the tech role for you know, over 20 years doing a variety of roles from dev to QA to Scrum Master to Agile coach. I've had the opportunity to lead engineering before, led product operations, and been chief of staff to CTOs and CPOs in a bunch of different B2B SaaS companies like Rally Software and Carbon Black, um, SignalFX, to, to name a few. Uh, I've also been on the investing side of the house, so on the other side, and at a place called Insight Partners where we would consult with a bunch of portfolio companies and talk about things like we're talking about today. So today, right, but currently kind of I'm running R&D Ops, Chief of Staff to our CTO, CPO at Clary. I have a TPM organization that supports engineering and then I have a small product operations team that supports our product management team. We're responsible for running all the strategy, planning cadences, you know, measuring and reporting product performance, you know, efficiency stuff, team health, all the way from voice of the customer programs and rollout programs.

Yeah, so lots of experience and a lot of different circumstances. So let's, you know, just to dive in, let's talk about a little bit about your situation at Clary right now. So you joined a little over a year ago. Where was Clary on that delivery predictability scale?

Yeah, well, let's make— let's say a little bit of, uh, kind of what Clary is and where it's at. So, you know, Clary's a revenue intelligence platform. You know, it facilitates and sort of drives, uh, the revenue process for companies. So forecasting, deal, deal inspection, opportunities management, things like that. We also have another product that's a conversational intelligence product which is called Copilot, because who doesn't have a product called Copilot in their portfolio? And then we also have a sales engagement product called Groove. Uh, we compete with like Gong and Salesloft and Outreach, just to name a few. Clary has about 700 folks. Our R&D organization is around 300 people, 50 teams, and then we have 3 sites, uh, US, Poland, and India. So with that background, I started in Feb 2023. I think, you know, Clary is your classic kind of scale-up company, was rapidly growing, hiring a ton of folks over those previous 2 2 to 3 years. They did an M&A, that Copilot product, in the summer of '22. And just things, you know, as you can imagine, scaling that fast, things are a bit chaotic across people, process, and technology. We saw lots of dependencies, things— we were lacking transparency into delivery. We were building things we weren't finishing. I'm sure this is just like— people have the same thing happen. And, you know, we just lacked overall focus, and that kind of eroded a bit of trust with sort of our internal stakeholders. Late in 2022, we hired a new engineering leader and his whole focus was just get his arms around everything, try to make us more reliable and predictable as an organization, not only for our internal stakeholders, but obviously for our customers. And we started to think about this in sort of a, a three-step approach, which was first, like, let's focus on predictability. And then second, let's focus on the efficiency of the organization. And then lastly, we'll start to focus on being more effective in there. So predictability plus efficiency is going to lead us to be more effective. And then how can we kind of even, even dial that up a little more?

Yeah, so I think that context is really helpful. And I think a lot of people in the audience are going to relate to because you're in a really complex environment, right? Like you have multiple locations, you've got acquisitions coming in, like this is not a simple business by any means. So I think, yeah, just learning how you even went about navigating that is going to be huge. And I think, again, problems of lack of focus. Are we following through on what we committed to? Getting that visibility seems to be really key. So really curious, how did you go about setting up a system to measure delivery predictability at Clary given, you know, that complex environment?

Yeah, I mean, so when I joined, it was, you know, just, it was super hectic. We did an assessment, right? It was sort of a qualitative quantitative assessment across people, process, and tools. I think it was super clear that we needed to start with process. We just saw everybody was doing things differently. We weren't shipping all the things. It was, so we took that and then we started to think about shaping how we were going to kind of, kind of start to build this system. So first we started to balance the teams. We wanted to make sure they had what they needed to deliver. They were cross-functional. We found spots where like we didn't have enough QA over here and we had to move some people or engineering managers' span of control wasn't as big as it should have been as senior managers. And we started to fix all of that. The next thing is we trained everyone in Scrum, right? In India and in US, we put them all through Scrum training just to get to a baseline, same nomenclature, same same understanding. When we said backlog refinement, everybody knew what the hell backlog refinement was. We then rolled out quarterly planning, right, which is— I call big room planning, which started at Rally Software. And that meeting was— we, we flew everyone in person. So we flew everyone into the Bangalore office in India, we flew everybody to Sunnyvale. And about that time, we had about 30 teams, and we started to plan a quarter's worth of work. So laying out kind of in a waterfall fashion, 80% Sprint 1, 70% all the way down to getting you know, a 70% commit on the quarter there of capacity. We made sure we had a prioritized set of initiatives across product and architecture. That was very difficult, but we got to a list. Then, you know, in that event, we could manage those dependencies between teams, right? Someone could walk right out of the room, conference room, to the next one and say, hey, are you going to do this? Yep, I'm doing this sprint 2, sprint 3. And we could start to stack all of that stuff as we go. We could then track risks and elevated risks in there and do that. So very transparent. People could hear what other what people are working on and go, oh, that's interesting, I didn't know you were working on search, and what does that mean to me? So a lot of cool things happen there. Then we, we really put an intense focus too on Jira hygiene, right? We created a work breakdown structure, initiatives, feature story. Everybody knew what they— what those were. We made things like negative query dashboards where people weren't filling out the right things in Jira that they needed to. We can now track sprint velocity, right, sprint over sprint. We could create security, you know, vulnerability dashboards, quality to escalate, We had everything. So we made sure we put everything into, as at Clary we say, into the CRM, right into our CRM, which is Jira here. Right. So, and then lastly, like I think the big thing was creating that feedback loop, which is we implemented weekly steering, right? Teams would come in and talk about their plan. Were there any new risks? Are there any new dependencies? Are you shipping anything big? And then the other thing that we would do is we would have a forecast in there. So they signed up for a payload over a quarter and we would say, hey, what's the forecast? Are you going to get done? and we could see that maybe you're not going to get done by the end of the quarter. And what are we going to do with scope? And people would have to come and talk about that. And then lastly, bigger, we implemented sort of a say-do metric across all of our teams coming out of quarterly planning. You signed up for these features, did you do them or not? And I think that we started with a target around 70% of that across all of our teams of teams to allow for some stretch and not, you know, just sandbagging, so to speak, and like kind of start to build from there. But now we actually knew if we were starting and starting and stopping something, right? So with that metric, we could drive to getting to the finish line more, right? And I think those handful of things have really started to get us onto that predictable journey and that predictable system.

Yeah, no, I mean, there's so much good stuff here. I was taking so many notes. Well, I guess I can't see. What really stood out to me here But one, I think, you know, looking at the team topology and seeing like, are the teams set up in a way where they can succeed in this environment? I think like that, that was huge. Another piece too, like getting everyone on the same page using the same language. I feel like that's a common pain point that we hear. So I think that is like super good to call out. And then what I also love too is that y'all brought people in person for planning, right? So that like shows, hey, like this motion, this is a business priority. We're going to invest money in bringing people into the same room so everyone could get the same context. I really, really love that.

Yeah, we want, we want people who are going to sign up to deliver that plan to be involved in building that plan. Because they've got to go make and meet those commitments, right?

Yeah.

Yeah.

And I just like bringing them along for the journey too, like being a part of the process, getting that buy-in. It's really important that people doing the work understand it. That's right. And then another thing I love too is it's one thing to set this quarterly plan, but then how do you drive accountability towards it and know, are we even good at planning, right? So like building in that feedback loop with the steering meeting.

Oh yeah, it's great. I mean, it's great. And I think SayDo has helped us too. So just being transparent, having everything in the system, anybody could click on anything and see it is just like it's the first step to that predictability system that you need to build.

Yeah. And then I guess lastly, before we move on to the next point, I love too the flexibility in that SayDo ratio, right? You know, that's 70%, that's saying like, hey, we can still react to market needs or like a changing market. Like we've got flexibility to adapt, right? Where that 100%, you know, it seems almost like a scorecard and that could be, you know, challenging for teams.

I had to sit and shift that with our CEO. Like, why isn't it 100%? That doesn't— no, no. So, but yeah, it gives some space for us to be flexible and I think he got it after explaining that.

Yeah, that's awesome. Well, sounds like y'all put quite a few systems of measurement into place, and I love that these steps too, they're all adding up to providing visibility into like what was hard to, you know, sense previously, not only for leadership, but for the teams themselves. Like everyone understanding how do I fit into this bigger picture? So when you put in these systems, like was it the impact you expected? Did you receive any pushbacks or road bumps?

Well, you know, I mean, I think things got a bit more hectic for us, right? We, we did another M&A in that summer of 2023. That was our Groove, our sales engagement. We hired a new Chief Product Officer shortly after that, so a new executive showed up. And then we even opened an office in Poland, right? So we've now gone from one sort of product line to a suite of products. We rolled our Copilot product into the process in October. Then we added Groove just this year in February. And, you know, like I would say, luckily we had that playbook above that we talked about, those, those handful of things, because we put them through the same thing, right? So now we're— some folks are a little bit more mature, but, but we're all starting from the same, same home base to be able to, to kind of evolve. So allowing that with that playbook allowed us to start sort of measuring and forecasting at a couple different different levels, right? So the team level, right, supply and demand around capacity, uh, and, and, and also how much are they, they shipping through velocity. Then we could get the teams, a teams level, because we started to then say what's the throughput of how many features are getting through the system. We also had t-shirt sizes where we could say, okay, we can balance this, it's 2 to 3 features, and now we know sort of that supply and demand, which has also been super helpful to, to get to be a better predictability. So That then jumps us into now we can start to build like a 3-quarter rolling roadmap. So now we have a bit more confidence in, so we could actually feel pretty good and maybe 2 quarters out and say like, look, we've got some data that says we can meet these. And, and we start to get a little bit more confident and predictable in building, um, building all of that trust with our partners in marketing and sales for launches and different things like that. So Yeah.

Yeah. And I love too the playbook aspect, as you said, like, you know, you put the process in place, you did it with a smaller organization, figured out what worked, what didn't. So then as the organization grows, right, you can adapt and evolve the process. But like having a place to start sounds good. Not having to reinvent the wheel every time.

Nope. And having everybody go from that same place and then evolve from there is an important piece. Yeah.

And then also, I think, as you said, you know, starting with teams as the baseline unit and then building up to that and seeing like, okay, can we have a system or a way of measurement that's consistent across teams first and then layer that up, which can be a challenge for sure.

Yeah. No, no, no. And I think, I think, I think just a great culture at Clary to accept some of this change and to move in. Everybody's a little like, it's change, but but it's actually helped us to get better. Right. And people see that, you know, if you show up every day, you kind of keep doing the same thing, you're going to see those results. And I think the team has seen that and is appreciative that we're now tracking that stuff to be able to have a conversation to say, no, no, no, no, that's too much. You're giving us too much. Here's what we can do. And then we can have a better conversation in those, in those areas.

Yeah. And having a data-informed decision, especially with, as you said, your counterparts and marketing or other parts of the business that are reliant on y'all That's right. You're saying, hey, no, we have a data-backed process. Like, we're doing this based on our past performance. We're not just, you know, fluffing the numbers here.

Yeah, for sure. No, it's been great.

So, so what's next? I mean, how, you know, has your process changed more as Clary's grown?

Yeah, I mean, process is ever evolving. I think, uh, now that we walked into this new fiscal, like I said, we're sort of a true portfolio product. So we now are thinking about initiatives that cross product lines. So maybe we were trying to sell our Copilot product with Groove and we want to do cross-sell opportunities and there's work to do in there to make that happen. So now we have these initiatives that cross. So now we've started to think of, we've laid that base of teams, how we measure supply and demand and capacity, teams of teams. Now we're thinking about those portfolio-level conversations and what we're doing. And so we've essentially been trying, trying out just this year of like, well, how many teams does it take to deliver this larger initiative? And then we have some canonical model of teams of like, well, maybe it's a— I need an app dev team and one data platform team, both for a quarter, right? So now we have how big is it with, with a cost perspective and then how long is it going to take? And now we can start to watch that too through our system. And we can watch the throughput of initiatives. We can see if that's changing and how are we meeting those to then have some more data-backed decisions at really that portfolio layer and the capacity of the organization. I think the other thing we've been experimenting more with are dates. And I know everybody might went, "Shh," when I said the word "dates." But, you know, I think we've all had to do that for things like maybe a conference called GLOW. We had to deliver some new features, and we needed to put some dates on Teams. But we've been going a little bit further than that and trying trying to say like we need to roll out our new data platform to different cohorts and we need to get to 50% of our customer base and then 75%. So we've been building these milestones, which are a collection of features to get to those things and tracking against it. Um, I think, I think the other thing we've tried to do is we've tried to also introduce new products in, into the portfolio or new pieces of where we want sales to carry quota. So if we're going to ask them on the second half of the year to carry quota, right, on a thing that we're building, we better meet that date or forecast is off, our ops plan is off, all the things, and we start to lose trust. So we've been doing a little bit more experimentation there. Also, we've been testing the water with planning out really our bigger initiatives 2 quarters out. So like kind of looking in that space and saying now that we have a bunch of data on how how we've been building and what we can deliver, can we push that out a little further? But, um, so we're trying some of that. It could be different fidelity. Some teams might have to go a little bit deeper. Some teams can just use features and lay them out and know what their capacity is on the quarters. But sometimes you might have to go a little bit deeper there. We left that up to the teams and what we're doing there. So, um, biggest thing is we created just so much transparency across the company and with our with our customers with these new systems that it's just like, it's bananas, to be honest, from where I started in February. And internally, the other thing that we've started to experiment with was we have our big top 10 initiatives that cross our product lines. We've started to create a deck that we get, we send that out to the whole company so the whole company can see where are you at with these milestones, where are you at with these initiatives? it gets reviewed at eStaff, so they see that, and if they have questions on big things that are coming out. So we have this sort of interlock at many different layers of the company now, and just showing all the data that, that we have in the system that, that we're running. So, um, you know, I think setting expectations is a big part of predictability. Um, so, you know, as we transition into year 2, like I said earlier, we're going to start working on some of those efficiency pieces So there's a lot of new, like, that we're starting on dev productivity work around our build system or testing systems. And, you know, since efficiency is our next play into year 2 here, and I guess, I guess I won't go any deeper because I think that's going to be our talk for next year, right, Alexa?

Yeah, absolutely. I'm already committed to do another one next year.

Definitely got to follow up on that.

Yes. Yeah. So I mean, again, a lot of things jumped out, but one thing I'd love to follow up on too is, you know, again, the elephant in the room, dates. So do you have any, you know, pro tips for the audience on how you got teams comfortable with those dates?

Yeah, I mean, I think like we said, just like in quarterly planning, people build their plan and commit to it. There is no, like, we're going to say we think we need to ship this by 7/31, but you all, can you all validate that and build the plan to do that? So it's a give and take. It's not all top down. You got to meet the thing. It's a negotiation process, which is good. We also try to use data like scenario planners and different things like that to be able to get us to those places. So it's not all bad. I would tell you probably 5, 10 years ago, I probably would have been, this is really bad. Really bad, but like it's actually driven probably better conversations. It really has kept us even more focused on what's the most important things. So when we need to have trade-off discussions, we can do that and go through, through those motions, right? So it's— you got, you got it. Like it's an up, it's a down, and then it's a meet in the middle, and you got to build it bottoms up. You got to like kind of meet in the middle there with this stuff.

Yeah. And I'm, I'm glad you clarified there because I think You know, it, it's a touchy, it can be a touchy subject for some folks. And I think, you know, saying that it, it is a negotiation, right? And again, using the tools at your disposal, the data that you have to really come up with something that's realistic and achievable, um, based on the requirements or needs. And then also, you know, dates are kind of a forcing function to figure out, is this really important or not? Like, are, should we actually be doing this? So I think. In terms of using it to facilitate conversations around priorities. I think that's, it's really great for that. Yeah. And then I guess one other thing I wanna mention, I love too how you said, hey, the top 10 initiatives, we're sending that deck out to the entire company, not just, oh, this is just a deck or comms that goes out to our leadership team. Like, no, everyone has visibility, everyone knows what's being shared. I think that's awesome. I don't know if that happens everywhere, but I really, really like that. I think that's awesome. Yeah. So, you know, it sounds like y'all are well into your delivery predictability journey and have found a lot of success, which is awesome. You're like seeing some of the impact you really set out to achieve. So from your perspective, what was critical to implementing this system and, you know, critical to also getting buy-in across the organization? For doing so.

Yeah, yeah. I mean, I think, I think of a few ingredients that, that you kind of need, which is one, you need to create those long-lived cross-functional teams. They're rightly sized with the right spread of experience, right? You don't want to have, uh, 4 junior devs and, uh, a mid QA and, right, like you want to have a good spread of folks on there and make sure you're building teams that can deliver on what we're going to give them. From soup to nuts, right? Teams and then teams of teams need to have a common way of estimating and forecasting, right? So we need to know how they do it. Are they going to meet it or not? Next, I think we talked about this earlier or two, which is teams need to build and commit to their plan. Nobody's going to tell them what their, what their plan is. They need to go build it. They need to commit to that. You also, I think, need to have common cadences across the organization. Organization. That's not only within your R&D group to manage dependencies, but like, you got to be able to know we're doing a monthly newsletter and what's the interlock with, uh, PMM on that, or what, what's our cadence and how does our cadence align to the product launch. So you got to have that with internal and external stakeholders. Uh, you gotta have feedback loops to adjust, right? Those are the steering meetings, the metrics reviews, the dashboards, right? You— and then two last things I think about is that ruthless prioritization and that focus, right? You have to have that. Your leader's got to buy in. You got to get to the list. What's the list? Sure, the list can change tomorrow. It's okay. Do you have the pro— do you have the stuff to be able to steer things in? Do you have the right process to do that? And then transparency into the system. We talk about that, right? So I always say, you know, 99% of the time, um, if there's a problem, it's usually a system problem. It's not a people problem, right? Uh, and I think at Clarity, people understand that, and it safety to folks to raise that red flag early, right? Nobody shows up to work and says, "I'm going to do a terrible job today, Alexa." Like, nobody does that. There's something probably in the system that we need to go fix. So to me, the holy grail is understanding supply and demand in conjunction with making and meeting those commitments. That'll absolutely delight all your customers and stakeholders as long as you're putting the right thing into that hopper. But doing that creates that great, great system of predictability for you and your company. In my mind.

Yeah, that's, that's awesome and really well, well put too. And I think, you know, I guess, well, to wrap things up for the audience, I actually see there's a lot of, there's a lot of questions in the Q&A, which is awesome.

Oh, um, cool.

I'd love to hear too, like, has Jellyfish helped you in this process at all and how so? Anything you're willing to share with the audience today? I'm sure.

Yeah, I mean, I think we put our milestones now into Jellyfish and we have a target date, so that's linked in that top 10 deck that we send out to everybody so everybody can go in and look and jellyfish at things going. We look at like steering meetings, we did sprint carryover, that was always the thing, like our CTO was like, hey, why is that still 3 sprints out and you're carrying that thing over? So that would happen a lot. Obviously, we use the investment stuff for our board materials. I just did that this week with all of that. We're starting to move our team health into the DevEx capability. Capitalization is huge, also a big thing going on this week. So all of that stuff is helping. I think next is is getting into that efficiency side, which is our dev productivity metrics and how are we looking at that. But that's what this year is, is all about.

Awesome. Well, thanks for, for sharing those, those pro tips. And I guess like one thing to wrap up before jumping to the Q&A, like I really wanted to call out, you know, what you said about transparency into the system and how, you know, if things aren't going as expected, right, it's not about the people, right? It's taking a look at the process and saying, okay, what in this, like, system needs to be fixed? And I think removing people out of the equation really, really helps, like, build that delivery predictability engine. So, um, I thought that was a great call-out at the end as well, just making, you know, making it a safe space for folks because maybe historically we haven't been measured as much. And so even the idea of being measured is kind of scary. So knowing that there's a safe space case for that and that it's, you know, for, for the greater good of not only the business but like myself as an employee and making sure I'm doing, working on the right things and prioritizing accordingly and doing them well.

So yeah, yeah, it's all about the system.

So, uh, we have quite a few questions in the Q&A. I know we only have like 3 minutes here, so, um, you know, scanning There was one about, you know, just some clarity on how long the planning process takes. So the question is, how long does the planning process take? How do you balance starting so early that by the time you arrive on a plan, priorities have shifted, but not so late teams are compressed to engage in the back and forth negotiation? So it sounds like some logistics or questions around logistics there.

Yeah, I mean, I think you could take this a couple different ways, right? Like usually companies have annual plans and you got you want to kind of lay out the year so you kind of have this place where you're going. And then that needs to be refined quarter over quarter in my mind. So that's when we drop that in, look at that. And I think, you know, you're going to be executing while you're also planning. I think that's a hard pill to swallow for people, but eventually you got to get there because if you're just sprinting, you're not, you're not able to go out and make and meet commitments to your customers and then also to your internal stakeholders. So You know, that's the way that we think about it. We build sort of an annual plan, we check on it on horizons, and how do we lay that out quarterly and plan quarterly and check in on that through our big room planning process.

Awesome. Well, thanks for providing clarity on that. And we have 1 minute, so one more maybe. So it says, when you have a team that misses targets, who steps in? How do you handle that?

Yeah, so it's funny because since we're using more dates now, we just sort of build a little bit of a, you know, use some old school language, like a change management process. So we have an initiative team, and if they're going to miss a target, they know what their target is, they get to bring that up, and then they get to say, here's the 3 options we considered, we're going to go with this option. And then we discuss that in our PDE review meeting every 2 weeks. That where we look across those things. So they can bring the plan. We're starting to roll that out to make sure, like, if you're going to miss this during, like, what is it, why, and how do we do this? And if they need to escalate, right, like, we can get sort of CPO, CTO in the room and talk through it if we need to be.

Awesome.

Well, we've, we have 20 seconds left, so we're gonna get, like, pulled off, aren't we, too? It's like You're done after 20 seconds, right?

So, um, but thank you so much, Andrew. This has been amazing. I'm so excited to see, you know, the customers engaged too in the chat here. Clearly a lot of interest. So thank you so much for sharing all your knowledge, experience with the group today. It's been, been great.

No, thank you.

Appreciate it.