The Software Engineering Intelligence Maturity Model

Hi everyone, thanks for joining us today to talk about the SEI maturity model. I'm Ryan Kachova, SVP and Field CTO here at Jellyfish, and I'm joined today by Adam Ferrari, a serial CTO, board member, actually one of Jellyfish's first customers, and and now an advisor to both the Jellyfish and our customers. Adam, quick introduction of yourself and your background.

Yeah, thanks for that. Really super happy to be here. Happy to be here with you all. Like you said, I was an early customer of Jellyfish. I've been an engineering leader for a long time and just generally interested in the metrics space for engineering. You know, I've been a fan of the product and the company for a long time. It's really awesome to be working with the company now and helping, helping, give advice to our customers and internally. Also, just fun to be here at GLOW. I've been a many-time participant as a customer, and it's nice to be kind of getting behind the scenes.

Yeah, we're excited to have you. And today I'm really excited about picking your brain a bit, you know, as we dig into the model. Before we do though, we've gotten some questions on how we came up with this model. The model was actually created from data from almost 1,000 companies, over 100,000 engineers, experience working with our customers on this challenge for the better part of a decade, and then real-world practitioner experience, you know, people like yourself and the exact type of challenges that prompted our founders to form Jellyfish back in 2017. And the impetus to create this model was actually in response to increased pressures for engineering leaders to defend the productivity, if we can use that word, of their team, brought on by the fusion of both economic pressures as well as the emergence of AI. Our framework focuses on 3 core components. Is R&D focused on the right work? Is R&D aligned to do the work right? And is R&D doing the work cost-effectively? Adam, given your experience with other models and frameworks, how would you characterize this model versus those? What is it you do like about it?

Yeah, I'm really happy to see this coming out of Jellyfish. I've, for a long time as an engineering leader, I was always kind of envious of other functions where, you know, there's more established measures that they can use. As engineering often it's hard to show to your board and to show to the rest of your exec team like a quantitative way of how you're performing. But it's so important that you do that. So, you know, a long time, you know, tried to DIY it and it's been great. In recent years, there's been standards, DORA and SPACE emerged. But, you know, in my experience, there's still a problem in terms of getting those emerging standards to connect for your stakeholders because they tend to be a little bit engineering-centric and a little bit academic. I think what I see coming out of Jellyfish is it's a little bit more of a balanced, pragmatic take in terms of consumability. Also like the applicability of the model internally to manage to. But, you know, like you mentioned, that's the experience of the founders was as engineering leaders having to do exactly that, bring, you know, measures into a board and to try to explain how they're doing. So I think this captures that essence.

Yeah, that's great to hear. And what I'd love to do today with you is kind of walk through the model at a high level and pick your brain, right, as a practitioner, right, from your experience, have you seen this come to life and what types of trends that you're seeing? So let's dig right in. You know, the first one here is, is R&D aligned to do the right work? And, you know, we've talked about this a bit in the past and it's super difficult because a lot of engineering is actually a bet on future outcomes, sometimes purely experimental in nature, and it makes it really difficult to understand if you're doing that right bit of work. And I think people have struggled with this and, and suffered the implications of focusing too much on like ROI, for example, um, where they try to project a specific outcome. And there's, there's many shortcomings of that. One is like stymieing growth or even like innovation or speed. And others of that, things that don't have a hard ROI, oftentimes things like technical debt actually get deprioritized, right? So our model focuses on something different. It's actually around I'm going to say like the core behaviors that suggest that you're going to work on the right work. Right. And those are things around, you know, is engineering developing a roadmap that's aligned to the company and product strategy?

Right.

Do they implement the governance practices that help them prioritize work? Do they drive tight innovation cycles? So, Adam, like having walked maybe too many miles in the CTO's shoes, like trying to articulate these types of things and talk about the ROI or the impact of engineering and projecting that we're doing the right thing, but we're basing it on some bets. Like, how do you think about this type of challenge in your perspective?

Well, this, this was— I mean, right off the bat, this was probably one of the most resonant parts of the model for me. I think in the transition from zero interest rate, you know, one of the things we did see, like, you know, I know my board at that time, you know, came hard at the company to understand ROI and that we were operating efficiently. And, you know, your ability to project that you know, was so important. And in fact, actually, we worked with our largest investor at that time to build a whole ROI framework. And it's exactly what you're saying. It's kind of like, do you have hygienic processes in place to put a plan in?

You do.

You have to account for the different categories of investment and the different types of ROI, or you could starve important stuff. I think in the past, you know, when there were like the money was flowing, you could kind of leave it to faith that that stuff would happen. But like now, you know, you want to know that there's a measurable, you know, hygienic process in place to do the right thing and also to expose the knobs on that, right? Like, let's let's look at that in relation to our strategy and, and understand, are we planning the right balance to do the things we need to do in our product strategy?

Yeah, I'm a big baseball fan, and I like to think that, you know, 300— batting.300 in baseball gets you into the Hall of Fame. I think we can do better than.300 in R&D, but, you know, I think it's also a good reminder that you shouldn't be looking to bat 1,000 here, right? It's okay to make some mistakes, just make sure you have the right processes in place around them. But we think about this inside of the SEI maturity model here, We're focused on two components. The first is roadmap planning, measurement, and execution. And the second is R&D impact. Focusing on the first around roadmap planning, measurement, and execution. And this is really meant to answer that question around like, is the team able to develop a plan, build cross-functional alignment on a level of allocation against that plan, and then actually deliver it in a way that they build the trust of the organization. Adam, having been in that role and what you just mentioned about like the pressures that you were seeing around it, like what is some of the more important ways of thinking about this? You know, is it the type of work? Is it the level of specificity by which you plan that? Like, what are some insights that you would give?

Yeah, this is one where, you know, I think there's often a temptation for exec teams to want to get into the details of the roadmap. I do think, in my experience, thinking about it categorically is really positive. And in fact, actually, I mentioned we built that ROI model, you know, back in the 2022, 2023 timeframe. And that was a centerpiece of it was understanding the allocation model.

Why?

It gets a level of granularity that the exec team and the board can understand and, you know, kind of gut check. Does it seem right? And then you can dive in, you can give the anecdotes or the examples, but at least you've got a kind of a high-level consumable way to think about how you're investing and does it seem right in relation to sort of the product strategy stressors that your particular company is seeing.

Yeah, that makes a lot of sense. And what we're seeing are our most productive organizations, they're building that cross-functional alignment on the different buckets or types of work versus a specific roadmap. And they're delivering on that roadmap within 5% variance, you know, in either direction. It's not meant to be perfect, right? It's like, what kind of buckets and roughly how much do we put in each? But that begs the question and moves us a bit towards like R&D impact, or said differently, it's just kind of that measure of ROI, right? And in that world, what we're recommending in our model is that you look at R&D as a percent of revenue, but you only do that in comparison to growth rate. And we're finding is that the most productive engineering organizations are able to drive a growth rate that equals or exceeds R&D as a percent of revenue. You know, we like this because it looks at this idea of R&D as an investment and tries to get people to think about you have to spend to, to really grow. But, and put the onus then on R&D leaders to build an org, make the right bets by which they're capitalizing the, that they're using the capital efficiency correctly, really. To make sure they're deploying that the right way, you know, balancing tech debt, right? If you're going to grow fast on features but not minimize your spend on AWS or things of that nature, right, you're going to find yourself in a world where you can't maintain that ratio. So, Adam, when you think about this from your experience, like, how do you help communicate R&D ROI? What are your thoughts on this type of measurement or what recommendations do you have for folks?

I— this is clearly, you know, a critical ingredient to having a model. You know, I think every board these days wants to know that you're, you know, using capital efficiently and that you're investing appropriately. I think linking it to growth, that's, that's, I think, a nice innovation in the model in the sense that that creates a direct connect— connection between, you know, like what, what you are planning to deliver and sort of the outcome that it drives. Again, at a high level, that doesn't get into the detail of exactly how that growth is going to happen. But it's saying like, hey, we think this investment strategy is appropriate because of the kind of growth that we are anticipating and being able to, able to drive. And so it kind of creates a nice back pressure or a connection between your investment strategy and the outcome.

Yeah, I'm gonna build on that a little bit and this idea of balancing, right? And take us to our second core component around is R&D aligned to do the work right? And I use the word balance here 'cause I think this is where people often come into this friction of understanding, you know, can engineering be measured? This is probably where our framework probably overlaps with other frameworks that do exist in the market the most in terms of topic, but I think we're tackling it slightly differently. Right, where other models are often around activity or output focused, our model is really focusing on this idea of like outcomes. So Adam, when you think about measuring engineering, you know, what are some of the shortcomings of these activity-based metrics or output-based metrics as opposed to looking at broader outcomes?

Yeah, one, one of them we've already mentioned is like they tend to be a little bit activity-centric or engineering-centric. And so like they maybe they're not as relatable to your stakeholders. You also do get the finer grain. I think you get an activity, you get sort of more gameability if I'm like really focused on like, you know, how many commits am I pushing or, you know, how many— it starts to be something that, you know, from an observable perspective might indicate good performance, but it's not necessarily something you're trying to incent. So, it's complicated that way. So, I think focusing on the outcome, right? Like, A, that makes the model easier to link up to the rest of the company's business strategy. But also, I think it tends to mean that you're focused on the big picture and like how can you actually accomplish what you need to internally.

Yeah, and we think of that in the framework that we've been collaborating on here of having two components. There's the qualitative, you know, the things that you're actually looking to measure by numbers. Then there's the qualitative, right? Which is actually understanding how the team's feeling and getting their feedback on the processes because they know where the gaps are. On the qualitative side, there's like three legs to that stool, right? There's predictability, speed, and quality. And, you know, we've all heard the old adage that You can have something on time, on budget, or with high quality, pick two. But I think that while it does hold merit, right, it doesn't take into account the importance of trade-offs and managing scope. And I think with software development, we don't really know the end state at all times. There's a lot of trade-offs that can be made that can help you have all three of those. So I don't know, like, I guess in your experience, like, what do you, how do you think about the relationship between these three?

Yeah, I mean, it's funny, like, pick two doesn't cut it. You've got to have a decent balance. I think like the particular balance depends on the pressures that are— that you're facing. But you've got to be looking at all of these in relation to one another. There's no free lunch, right? Like, you know, if you're going to push on, push on, you know, one aspect of how you operate, then it's going to, you know, you're going to be taking resource away from something else. So it's like it's perpetually as an R&D leader, it's a game of tradeoffs. And, you know, you're always being asked to do probably a little bit more than is really realistic with the resources in play. So you've got to be really judicious about making those tradeoffs.

Yeah, that kind of brings up the, the qualitative component here, or developer experience, if you will.

Right.

If you try to squeeze too much water out of a rock on productivity, you can do so at the expense of your team's long-term productivity and effectiveness. Right. And you have to be able to balance that. And the second component of it is engineers know the biggest issues, bottlenecks, points of friction that are keeping them from doing their best work. Right. And we're seeing our most productive engineering teams actually scoring 75 or higher on what we refer to as the aggregate developer experience score. I'm curious, Adam, like, I know you've deployed some developer experience type, you know, surveys and/or analyses across the orgs that you've worked with historically. What has been your experience around that more qualitative developer experience sentiment?

Yeah, it's, it's funny. We often talk about the, even the name developer experience has a kind of a fluffy feel to it, but I, you know, I think at the end of the day, engineers are really pragmatic and they want to deliver outcomes. And so, I mean, for many years prior to this becoming more of a standardized thing, like we would always run developer surveys and that like honestly has been the source of not just actually focusing on the right problems. I mean, you know, there are periods I could remember where, you know, we had an allocation that we'd set aside for technical improvements, but the experience of the engineers on the ground were like, hey, we're not getting that time. Like, you know, we're getting too much PM pressure. It's not being able to allocate, like just being able to highlight that, right? Like, like this is something where it wasn't even a big solve. It was just putting a spotlight on the problem. But then also, like, you know, you get particular things where, like, you'll hear about friction in tooling or friction in terms of, like, you know, technical investments that need to be made. You'll not just get the problem, but you'll get, like, some of the best solutions come from the people that are closest to the work, right? And so, and I find that, like, it's not just a well-being thing. This is honestly about, like, if you want to, you know, mine for some of the best, you know, ideas, like, the engineers will give it to you. And I think this is, you it's becoming such a table stakes thing. It's really great. Like, I mean, I really appreciated seeing Jellyfish launch an offering here so that like it doesn't have to be custom because when you're surveying, right, you want to have good practices and you want to have consistency. So you want to have thoughtful design over time. And so it's kind of thing you don't want to be winging. So it's nice to see. I think it's, you know, really a table stakes thing in terms of a model like this. You have to see that pillar in it.

Yeah, I fully agree. And also make sure that while you're looking at higher level metrics, there's not some other issue that's manifesting, you know, right at the covers there that that's going to become a bigger challenge in the future. So it gives you the indicators. But you mentioned a little bit around, you know, making sure you deliver in the right way. That takes us to a third category here, which is around is R&D doing the work cost effectively? Right. And this is one of those topics that I think has always been a third rail topic because people have always addressed it in a way of like, what is your dev to PM ratio or what is your percent offshore or low-cost development, you know, and I think that takes this like really academic view of an engineering org structure. When I think a practitioner view is slightly different, right? And it thinks of capital efficiency across 3 core components. To me, it's like a broader capital efficiency, which we'll get to in just a second here. Uh, regrettable attrition. And then I think of it as like resource fungibility is the 3rd component here. What I'd love to do is quickly hit on each of those with you, because I think these are really complex topics. And when we think about capital efficiency in our framework, right, we talked about this idea of developing a roadmap and then executing against it. And we talk about measuring that originally in doing the right work around like percent of effort. In this bucket, we're actually calling back to that and we're trying to instead focus on spend, which answers that question of, you know, have you actually scaled your org in a way that maps to what the business needs for the growth stage etc. There's different components. Adam, how have you thought about that as an engineering leader, being able to articulate like spend by product or spend by type of work as a better proxy than maybe looking at just ratios, which is a lot of different academic theories and you need to have that technical and business acumen merged together to really opine on.

Yeah, again, like, like you have to have this, you know, like top-level examination of like, are you prosecuting a strategy that you can afford and it gets done what it needs to get done? Like looking at those lower-level metrics, like, you know, some of the drivers, like for instance, how much, you know, of low-cost development do you have? Or like, what are your ratios? Those are important. But at the end of those are kind of internal, like how do you get it done? At the top level, you have to ask yourself, hey, can we afford this product strategy? At least, you know, as currently we're prosecuting it and like, does that, you know, that then will marry up with like, are we pursuing the right product strategy? If it's not achievable, it doesn't really matter. You could have a dream, you could have dreams, but they have to be accomplishable within, you know, the dollars you have to spend.

Yeah, and I think you hit about dollars you have to spend. You know, one of the more direct cost savings elements is that the cost of replacing an engineer is often attributed to the— I think it's 2x the cost of the engineer is a pretty standard, you know, you want to think of it as a ratio in terms of spend there. So our second capital, you know, efficiency or effectiveness measures really that idea of regrettable attrition. And I think there's two lenses to this. One of them is that concept of, hey, let's have that immediate spend and savings, but there's also long-term utility to having tenured engineers that understand your code base and your domain and your customer base. You know, how have you thought about managing regrettable attrition knowing that our most productive organizations are able to maintain a regrettable attrition of less than 5%?

Yeah, I, tenure's always been super important to me. Like there's this legend of the 10x engineer, which I always think is a little bit of a, like a oversimplification and silly. Really what you see is like, you know, engineers that spend significant tenure in a problem domain, in a, in a, in a, you know, a particular technology codebase, right? Like you continually like increase in terms of your ability to reason about it, think about it, know, you know, the past decisions, what was tried, like you get more and more superpowers as time goes on. So it's like that 2x to replace hides like the upside that you might be losing in terms of like deeper and more innovative treatment of the stuff you're trying to do. Not to mention, by the way, like for me, it's always been like something that you wouldn't want to let snowball. Like, I think, you know, it can often be a lead. This is why this goes hand in hand with the DevEx surveying stuff. It's like you wouldn't want to let a little, you know, bit of smoke hide a fire that's, that's coming because these things can snowball if you don't, don't manage them and then you don't, you'll get that upside of the, of the tenure and all the, the long, long thinking brains that have been thinking about your problem space. Yeah.

And if I could pick your brain maybe on one last question there is You know, one of the things many engineers want to do in terms of like their willingness to stay at a company is getting new experiences, right? They want to build new skills, they want to work on new things. And so our last measure in our model here is this idea of resource fungibility, or the amount of time it takes somebody to ramp on either a new codebase or a new team. Our most productive organizations are really ramping internal engineers in about 2 weeks' time. Right? How do you think about resource fungibility and the importance of it?

Yeah, this is a, again, I think this is a really resonant part of the model for me. I very literally, like in my experience, I've, you know, this has been on multiple exec teams. One of the biggest frustrations actually comes from stuff we've intentionally done in R&D over the last, you know, decade plus that we really, you know, have strongly adopted these ownership-based teams, right? The Marty Cagan model, et cetera, which is awesome. I mean, it's, it's like you have these teams take ownership of an area, a part of the code, a part of the problem. And, you know, they build context and like they work together and they build trust and they really rock out in that area. And it's wonderful. But at the end of the day, like, you know, it's software and the market is volatile and it's unpredictable. And one of the frustrations, though, is like, hey, we know, you know, as a business there's these end things we might need to do, but it's hard ahead of time to know exactly what the competitors are going to do and what's going to resonate with customers so that, like, you know, you can't necessarily preallocate So there's some balance that has to happen and, you know, you need to be able to move people. And not to mention, by the way, I think as much as, you know, engineers love the consistency and being able to go deep on a problem, again, they're also very pragmatic and, you know, they, I think most, most would rather move where the action is at if that's what the company needs. So this is like, this is a really nice measure to me, very similar to like how some people look at just straight up onboarding. You know, fungibility to me is even more an important question for the business because again, you'd rather keep the engineers you've got that know the whole domain and the company and the history and all that stuff. Stuff, but you'd also like to be able to balance, you know, thoughtfully, not churn, but like you'd like to be able to thoughtfully balance. And that's really what the business, you know, in my experience almost always wants. So being able to measure that is, is I think both a measure of your internal hygiene of onboarding, but also what the business needs.

Yeah, that makes a lot of sense. And I would love to pick your brain and go way deeper on this, but conscious of time, you know, if anybody's really interested in reading more about the SEI maturity model, please download the ebook per the instructions on the screen. There should be a QR code and I believe a link as well. But please also reach out to discuss it further. Adam loves talking about this, as do I and others at Jellyfish. We'd love to engage with you, get your feedback, and talk with you how you can apply this at your organizations today. So thank you so much for joining us, and we will talk to you soon.

Thank you.