Robert Ta breaks down how Stripe, Uber, and DoorDash measure AI coding agents. Stripe's agent jumped PR throughput from 300 to 1,300 weekly. Uber saves 21,000 developer-hours yearly through AI review. But Jonathan pushes back: shipping more code doesn't mean shipping more value.
if you're a tech executive or engineer and you're trying to prove ROI of AI productivity using coding agents, this is some of what we've seen. And these are some of the metrics that leading organizations have publicly disclosed. Stripe, they have this Minions homegrown coding agent harness with model-to-task routing based on internal benchmarks. This improved their weekly PR throughput by 4.3x, so 300 to 1,300 per week. You could kind of— whenever I think of enterprise productivity, you know, cost savings is either your cost savings or it's I'm more productive and I'm actively participating more revenue generating activities. Feature velocity is the latter. So that's the way to sort of split this up in your head. So Uber GenAI gateway, single internal API service, teams cannot bypass it to call vendor APIs directly. This was in response to their token maxing through Q1 where they blew their budget and they tightened the screws down. So 90% plus of 65,000 weekly diffs are now AI reviewed and they estimate about 21,000 dev hours saved annually. That's a lot of hours. When I've looked at pricing hours for a feature, usually the benchmark I've seen is anywhere from 150 to 300 hours for a complete feature, whatever you want to call a feature. In enterprise land, the features are typically more expensive, takes a few sprints. So that's how I, I read this and I'm like, wow, that's a lot of features. DoorDash: internal AI code reviewer posts findings 7 minutes after a PR opens before human review. They're going towards the narrative of, okay, let's place the human judgment to a higher level task where it's higher leverage. So do have the AI with a good harness that knows our code base do a pass first, I'm guessing, as if it's a mid-staff-level engineer, then have the real engineer come in and, uh, use their own tacit knowledge and judgment. So this kind of brings down cost to $3 a code review. You can imagine a staff engineer getting paid $250K at least. That'll be cheap in the Bay. Uh, but if they got paid $250K a year, they their hourly rates at least $125 or so. Code review is probably going to take 2 to 3 hours. So I would say this is like 100x savings per code review and it increases the throughput of high-quality code going to the codebase. And they say 66% of code is now AI written, catching up to Cloud Code and their team at Anthropic saying 100% of their code is now AI written, I believe. So these are some of the ways I've heard of teams measuring ROI. I'll also just list out some other things that I've heard is a desire to measure reduced rework. As you know, me and Jonathan, we're builders. There's the great messiness of product building. Unknown unknowns creep up and you make quick decisions, especially when you're shipping fast, and some of that could cause rework. There was this desire from a customer we have around, okay, well, if we're doing this right, we should be reworking less, right? And our cycle time should improve. Other things to consider for an engineering team Is your quality bar still good? You're shipping more, but do you have more regressions? Because then if you do, then you're not compounding well. So I would really look at your test strategy and quality and automation strategy paralleling your output throughput strategy. One last thing is the PM to engineering handoff. How do we decrease cycle time from PM to engineering handoff? This is very engineering focused, but there's a question of if product points us to the wrong thing to build, then we're just shipping the wrong thing faster, right? So upstream of all engineering work is what is the thing to build? And we're seeing trends that product has got to keep up with engineering. And that either means they're picking up coding work or being able to code on their own, creating code-first prototypes to learn more about what the right thing to build is, to be able to have a healthy backlog of what is the right thing to build for engineering for their sprints. Jonathan, I'm curious, what do you think of measuring value in the AI agent coding agent era here?
I think like, you know, in engineering, we pointed this like PR output as like a sign of productivity. I think before AI, PR like count wasn't really considered an important metric because it like didn't tell you whether or not you were shipping value to customers or not. But every engineer that's worked with these tools know like you can get a lot more code out the door. You can get it reviewed. You can get it merged into main through these PRs. What I think is interesting from this perspective is that like you're not seeing the same story of role dissolution that you're seeing in the frontier companies like OpenAI and Anthropic describing how their product management teams are like writing as much code as the developers are fully, you know, sort of like the product management teams are fully AI-pilled, the design teams are fully AI-pilled. And I think, you know, like measuring ROI in an engineering organization has always been kind of like, just going back to the mythical man-month, it's been kind of an illusion, right? You can't really talk about ROI for a company just by drawing a circle around engineering and looking at efficiencies. It's not a cost center in that way where like, you know, if you're automating customer service and you can like remove a certain headcount, it's not what's happening on the engineering teams. They're just like getting more efficient at doing the work they need to do, but You know, I think the company-wide metric, like, is still, like, eluding us in terms of, like, how do these gains in coding return as ROI to the company as a whole?
