In this talk, Jake Archibald explains why relying on a single metric leads teams to game the system, causing unintended negative side‑effects. He contrasts measuring raw outputs (e.g., number of PRs landing in production) with focusing on desired outcomes (high‑quality customer features). By illustrating how a narrow output metric can reduce code review quality, increase bugs, and degrade customer experience, he shows the need for a balanced metric suite. He proposes measuring multiple complementary outputs—time spent on new features, PRs landed, PR comment count, and epic cycle time—to collectively drive the outcome of delivering high‑quality features while preserving collaboration and focus. The overall message is to design metric systems that incentivize the right outcomes rather than merely the easiest-to‑track outputs.
Chapters
[Speaker:JAKE ARCHIBALD] Regardless of what we measure, what we've learned here is the system is going to cheat. If we pick a metric and we say we're going to follow it, all of a sudden the system is starting to say, hey, I'm going to find the best way I possibly can to optimize for that one metric. So metrics stink, right? We should just never use them. Why would we ever focus on them? Well, there's actually a way to disincentivize the system around this. To disincentivize this cheating, what you want to do is you want to focus on the outcomes that you're actually driving towards over the outputs that you're measuring. And this sounds a little bit complicated, so let me explain with a couple separate examples. When we think about outputs, we could start to say we want to increase the number of PRs that we have landing in production. That's a pretty classical engineering output. Are we actually merging enough PRs that then get landed in production? Now that'll have two second-order effects, as we like to call them. Number one is we're probably going to increase the number of deployments because we're trying to land more things in production. And to increase the number of deployments we might have, we might have smaller pieces of code because it's faster to write smaller pieces of code. Those will land in production even faster. Those are all good things. Like, we feel pretty good about that. But on the flip side, we might actually suffer some bad second-order effects if we're incentivized and focus entirely on the number of PRs that we have landing in production. We're probably going to spend a lot less time actually reviewing software. We're probably going to spend a lot less time actually reviewing each other's PRs, giving each other meaningful feedback because we just want to get things out quickly. That means that we're probably going to have more bugs. More or less people looking at something means that it's more likely that we're going to miss an edge case or something along the line. And what that's actually going to lead to is a diminished experience for all of our customers. Our customers are going to struggle. with actually being able to use our software because our quality by definition has gone down. So how do we fix that? Well, imagine we focus this around outcomes. For outcomes, we want to focus on having more high-quality features for customers. And what that means is we measure several separate outputs that would actually drive towards this. Number 1 is we could focus on time spent on new features. The more time we spend on new features, the more likely it is that we get more high-quality features for customers. Just like we mentioned in the previous example, we could measure the number of PRs landing in production. Making sure these things actually get shipped out the door. But to make sure we counterbalance that negative quality example, we can start to measure the number of PR comments to make sure that our quality doesn't suffer, to make sure our collaboration doesn't suffer with our drive to get things out the door faster. And the final thing we can measure is to make sure that we're not scattered and all over the place. We can measure our epic cycle time as a team to make sure that we're actually focused on one thing before we move on to next, rather than being scattergun across the board.
