The AI Coding Revolution

Good afternoon and good morning, everyone. I'm Andrew Lau, co-founder and CEO of Jellyfish, and I'm excited to welcome you here to GLOW Live. We've got a great program for you today, and I'm so glad you're here. So let's dive in. For engineering leaders, there's perhaps never been a singular technology so wholly dominating this moment. Of course, I'm talking about AI. Putting aside the larger societal questions around AI's rapid evolution, it's safe to say the software industry as a whole has a new layer of pressing concerns. It is also clear that AI coding has already transformed the day-to-day work of software engineers, software development processes and best practices, along with cost assumptions in shipping and supporting software. As engineering leaders, I know you're acutely aware of this, but it's still too early what this will all look like when things stabilize. To understand how we got here and just how quickly things are evolving, let's take a look back just one year ago. 2024 was a transitional year for the software industry and R&D departments specifically. This is of course fueled by the adoption of AI coding tools. And it's not just me saying it. If you're familiar with the annual DORA report, in the 2023 edition, AI got one single page of coverage. Fast forward to 2024 and AI is the sole topic for the first 46 pages. That's a 128-page report. That's more than a third. That's huge change. I know the numbers. And I'd say that change in coverage actually maybe even under reflects the mental coding space, the mental space AI coding occupies for your average engineering leader. Take this for instance. DORA's 2024 report reflected the change in sentiment around AI with a massive uptick in prioritization of AI as well as adoption. The report shows 75% of developers are using AI in some form. These developers using it through writing code, summarizing information, explaining unfamiliar code, optimizing code, and so on. This increased adoption has brought with it a reported increase in productivity. DORA found that with that 25% uptick in AI usage resulted in a 2.1% rise in productivity. Hmm. I think it's probably more than 2%. 2 doesn't sound that substantive to me. Now we know this data, as exciting as it may be, relies on anecdotal evidence, self-reported information. For the amount that your companies and mine are investing and care about AI coding tools and AI in general, anecdotal survey data frankly just doesn't cut it. Which is why, in part, last summer we launched AI Impact to help you quantitatively measure and understand impact and adoption of some of the most widely used AI coding tools out there, including GitHub Copilot. You'll get to hear from those folks later today. Companies are investing so much in these tools, they need accurate measurement to determine ROI and how to best leverage them. Because the organizations that get the highest return on their investment and truly leverage technology are the ones that are gonna win, the ones that will get that yield. Not to mention, these tools are changing all the time. Shifting the way that people are working. The industry needs accurate measurement to know how to cope with these changes. And here's what I mean to say the space is changing quickly. GitHub Copilot, as you know, launched only in 2022. It's now used by more than 50,000 organizations, and that was actually January '24. It's a year after that now, so it's gotta be more than that. Look, Since then, dozens— I don't even think that word even captures it— dozens and more have entered, new players have entered the space. All of these emerging solutions are changing software engineering. To me, this presents a huge opportunity for our industry and for your organizations and mine. Even still though, we're in the early innings. Most companies are still in a proof of concept stage of these tools. But the impact is already real. A 6-month proprietary Jellyfish study of 20,000 software engineers across 300 organizations using AI coding tools found, drum roll, a 25% speedup in coding. So you're turning around coding tasks 25% faster. We're not talking about building throwaway prototypes. This is real teams doing real production and real engineering tasks. That's kind of crazy. And we found across this corpus that there was a 12% increase in throughput. Plain and simple, engineers are putting out more stuff. But of course, coding isn't the whole thing. There's planning, there's testing, there's documentation, there's reviews. But still reflecting the overall process, we're seeing 12% more throughput. PRs being moved, this is indicative of work being done. 12% is more than great. This is just the early innings. You might think this is modest, but just think about how this adds up across an entire organization. A mid-sized engineering team spending $10 million a year, you just got back $1.2 million, or $1.2 million more. I want more. The wildest thing to me, folks, is that this saw a 17% increase in roadmap versus KTLO. That's keeping the lights on, folks. Shipping new stuff that moves the needle, not just support and maintenance. AI can, when applied thoughtfully, help with the maintenance and quality. When that kind of work is done faster and better, we spend more time on innovation work, as we see here. And you're not just getting more of the same, it's more better stuff. AI can be a force multiplier. All the while, despite these obvious benefits, In that same study, again based on real usage, not self-reported anecdotes, shows that 35% of folks aren't even using it. So what does that mean? These improvements are great, but we've yet to really get the AI coding yields we want and how usage changes the bigger picture. What do I mean? Well, very few engineering organizations are yet making changes to their headcount planning on a forward-looking basis. They're not planning around the changes in opportunity here. And adoption, frankly, is still lower than it should be. We're already seeing productivity improvements, but full productivity as such remains not tapped. Look, I sound like a bear, but I'm actually a bull here because there's more to come. So where are we here today? AI coding tools have actually gotten pretty darn amazing. We're seeing success from casual vibe coders, I need a t-shirt, vibe coder, building prototypes all the way to seasoned architects doing real core work faster and better with AI. But the tools and practices are moving fast, so fast. We started with chat interfaces generating code, then we quickly saw AI integrated to IDE environments. And now we're seeing those environments get more powerful, capable with agent-style execution and generate, and execute even larger project plans. We're seeing AI move beyond coding into other development tasks like deployment and updates and managing infrastructure. Exciting stuff. And yet we're still coping with the limitations of AI, like hallucinations. We'll regularly see AI refer to a library that doesn't exist or actually exist, or use an older coding pattern that just isn't the way we do things today. Depending on who you talk to, you can find opinions that range from, "AI will always be limited to being basic, handy auto-suggest like Clippy," to other folks saying, "AI will replace us all, programmers wholesale." The basic truth is we just don't exactly know where this is gonna land. We also probably don't know what the best practice will look like in a year, 2, 3 from now, maybe in 6 months. But what we do know for sure AI is the biggest change in the practices of engineering we've seen in most of our working careers, and this change will keep coming for years to come. It'll have huge implications for how we actually build software in the future. So where are we heading in the future? Well, maybe it looks like this, but it probably isn't gonna look like this. So what is it gonna look like? Ooh, pretty, right? All these developments are really promising, like I said, and I'm a bull plus plus on all of this. But the technology impact is not yet fully baked. The vendors are changing, technology is changing, adoption, the way we use it is changing. You need to stay in the game on measurement so you know what's going on to manage these changes. So you can make those decisions in an ROI-driven way, in a data-centric way, and invest appropriately. Not over, not indexing for, like, not lower for the wrong day, getting it in right. We're going to be in that time of flux for the next few years, and you need to have that telemetry. That telemetry is essential so you can iterate and adjust accordingly. We don't know what the future looks like, but we know that we're here with you figuring this all out together.