The New Bottleneck: AI That Codes Faster Than Humans Can Review

You're listening to The New Stack Makers, a podcast made for people who develop, deploy, and manage at-scale software. For more conversations and articles, go to thenewstack.io. All right, now on with the show. Hey, everyone out there. It's so great to see you again here at Google Cloud Next. Extent. I had run across the website. I saw this company called CodeRabbit and I thought, huh, what are— what's CodeRabbit about? So I went by the booth and I saw Harjot. Harjot Gill, how are you?

I'm doing good.

Harjot is the CEO and founder, correct?

That's right.

And Harjot reminded me that we have talked in the past and we covered their company. Maybe you could tell me a little bit about those years, 2014, 2015, 2016, and what you've been doing since then?

Yeah. So, back in the day, like when all these microservices were taking off, containerization was taking off, and that's when I did my first startup actually. So, that was like fresh out of the research I was doing at University of Pennsylvania. And that company was called Netcel, listen spelled backward. It was like observability through network events, right? That was the first time I guess we got in touch with the NewStack and we got some coverage back in the day, and that company was acquired by Nutanix back in 2018. Then after that, I did another startup called Flux Ninja, which was in the reliability management space, looking at rate limiting, QoS, load shedding, and so on. And now the third startup, CodeRabbit, is in generative AI.

CodeRabbit.

Yeah.

And so can you talk a little bit about how CodeRabbit emerged? What were you seeing that struck you as a unique issue in the market?

So one of the things we saw back in when Copilot came out, like that was like the big aha moment that the AI has gotten to a point where you could actually build interesting systems with it, right? So that was the time I was in my second startup. We had a team of around 15 engineers. This was during the days of the COVID And one of the things we saw is like, The big bottleneck in the shipping software was the code review process, right? Where you had a human reviewer looking at pull requests on GitHub, making sure they didn't have any security issues, performance issues, we were following best practices, consistency, and so on.

Maybe you could back up. What has historically made it so slow?

The code review is usually where you need to have some sort of a senior talent to understand the implications of a code change on the larger system.

Ah.

Right? So the code review is usually like a practice which companies like Google follow really well. Like they have like really good senior talent who can maintain consistency and so on, making sure there are fewer security issues or bugs getting shipped into production. And the reason for that is like the sooner you find these issues, the less expensive it is. Like you're talking about orders of magnitude, expense when those issues are seen in production, right?

Right.

And almost every issue that you see can be traced back to some code change, be it a production deployment issue, be it a security issue. It always has to do with some code change, like most of the time, right? So code review is like a very critical step. Even pre-gen AI, it was a big bottleneck, right? So people had all these static analysis tools with a lot of false positive rates. They were using, I don't know, SonarQube, bunch of these tools, right, in the CI/CD. And then there's a human review step, right? So usually this becomes a very slow process. And in a larger company, we have seen— I was at Nutanix, right? My first company was acquired there. So once you grow your team to be like 50, 100 people, like at that point you realize that shipping every change takes a few weeks. So these pull requests, you have like a lot of ego clashes, a lot of discussion back and forth. And it's not like you can ship software within a few hours. It takes a few weeks in many places.

I see. So, Code review has been a thorn in for tens of thousands of developers out there, hundreds of thousands, billions.

Exactly. And it has become worse, actually, believe it or not, now with AI generating a lot of code. Like, people are doing wipe coding. That term is gaining popularity. And the idea is like you're starting with this high-level prompt and then AI generates a bunch of code for you. So inherently, the software developers are now becoming reviewers of AI code because a lot of the times you don't know what's going going on in that code because it is a lot of code. Like if you're using Cursor Agent or something, and that's exactly where CodeRabbit is seeing success as all this AI is taking off in code generation. The review was already a bottleneck, but it has become even worse with AI. And that's where we are coming in as a code review system to automate a lot of that process.

We talked a little bit about the roots of the problem that Originating from just traditional code review, right? How has traditional code review been automated in itself? There must be lots of advancements in automated code review.

Right. I mean, that's where the magic of large language models comes in.

But before that, there must have been automation efforts.

I mean, there has been some automation mainly around security rules. Like if you're familiar with like companies like Semgrep and You're familiar with SonarQube. So you're running a lot of these linters, static analysis tools, and these are rule-based systems, right? And the trouble there is figuring out the right set of rules which are not very restrictive and have a false positive rate. And usually people start with broad set of rules and they start narrowing it down. But each time you narrow it down, you have to clean up your existing code base to comply with that rule set. So it's complicated, right? I mean, to enforce those automated standards.

So to enforce those automated standards, it gets difficult with rule-based approaches. Rule-based approaches have historically been very difficult to implement, haven't they?

And that said, these rule-based systems do not understand the intent of the code change, right? So when the human comes in, they're looking at the architecture, they're looking at the implication on— they have a lot of tribal knowledge, right? These senior engineers. So they're not just looking at lines of code or the syntax issues and the coding standards. They're like going beyond that. So they're bringing in a lot of knowledge on the best practices. So that's why there was always like a human review step despite having automation as well. So automation alone also was never successful.

So you recognize this problem and you've taken an approach in what way? What's the approach that you're taking?

Yeah, I think the magic is with the large language models, right? So especially the reasoning models. So CodeRabbit happens to be one of the biggest users of reasoning models in the industry, all the way from O1, O3 Mini, Sonnet, 3.7. We use them all, right, at scale.

How do you use them?

So what we do is like essentially our product inserts into the pull request model. Like if you are on a Git platform like GitHub, GitLab, Azure DevOps, Bitbucket, right? So as your developers write the code in the IDE and now they are ready to ship it, shipping means now you're ready to merge your change into the main branch. So that's where you open something called a pull request.

Right.

Right? And in that pull request, bunch of other things happen, right? So you are running your CI/CD pipelines, you're running unit tests, you're running analysis tools, static analysis tools, and then there's a human review process. That's where CodeRabbit comes in. So it looks at the code change. It's an agentic system, first of all, right? So it looks at not just the code change, but also the implication of that change on the rest of the system. Then after analyzing the intent of the change, it provides very human-like feedback in form of review comments, just like a human does.

So just to back up a little bit then, a pull request gets made.

Right.

That's when CodeRabbit emerges, right?

Right.

And I'm trying to trace it back to what you're saying about how CodeRabbit is one of the biggest consumers of the various models that have been developed. Is that because you need that context?

It's about the context. Yes, I think.

Is that why you're consuming all those models? Because you need to have that context to better to like kind of fit into that role of that human in the middle?

That's right. So these large language models are impressive by themselves, right? But to really make them effective, The big difference in like the startups or the companies in the AI-native startups that are doing well is the context, right? And context comes in many ways, right? So we are talking about context from code itself, from the diffs and all, like from the entire files. The context comes from the code graph, how that file's code is related to other files. So you're doing the code graph analysis, right? The context comes from learnings that users have provided from their past interactions with CodeRabbit, right? So context comes from like many, many, many ways, right? And that entire context is used then to understand the changes and the implication. And the other thing is like we try to like be proactive with the context we bring in, but sometimes it's not enough. So in that case, we also give AI, the large language models agency to go and find information that's not there.

How do you measure that, whether it's not enough? Because in human terms, you're trusting the individual who has seniority to review the code.

Yeah. So this is where like the large language models, you have prompted them in a certain way that, hey, if you don't know the answer, you can do these things like do some tool calling, use an MCP, for example, like which is taking off to get context. Like RAG is one approach of bringing in stuff into the original prompt. But then after that, If the large language model decides that, hey, I don't have enough context to make a good inference, but I have all these tools on the side that I can use, by the way. I can go to Jira and pull in the issue information. I could go read another file, right? So you give all these tools to the AI to go and find those answers and bring in additional context during the review.

So that's what you're doing. You're giving additional tools to the AI.

Exactly.

What are those tools that you're giving to the AI?

So one of the things we do during the review is we create a sandbox environment, because we are a central tool. We are not in the IDE, like Cursor and others. The code is not there right now. So first of all, one of the things that CodeRabbit does is create a sandbox where we clone the repository, the entire repository, the entire codebase, not just the changes, but everything. And in fact, I have a talk here tomorrow at Google Next, talking about how we do that at scale using Cloud Run. One of the things we do there is like in that sandbox is we bring in all the context, which is the code. We also install some dependencies like which we need to understand that or run the software and so on. Then AI is given access to that sandbox that here you go, you have this computer at your disposal. Now go and navigate it just like a human does. At that point, AI can generate things like CLI command shell scripts. It can use cat command, for example, to read a file. It can use keyword searches using ripgrep, right? It can use astgrep to read functions, right? So, we've given it like a lot of these CLI commands at its disposal to just navigate the code like a human does and get additional context if it's not been there from the beginning.

So, it just navigates the code like a human does. How do you then create that environment to— in the research that you've done to have it behave like a human does?

Yeah, that's where a lot of the Usage, one of the things where I think CodeRabbit was successful was we made the product initially free for open source. And there were 2 reasons to do that. One was, of course, it helps with marketing, but the second was just user behavior that the insights that we gathered from usage actually helped us do the right kind of prompt engineering to some extent and run the evals. Like, I think evals are pretty critical here, like, to make sure that you're guiding the AI and nudging it in the right direction. For example, I'll give you an example. Like each time we see a package.json file getting updated with a new package, so we have prompted our agent in a way that it goes and looks at security vulnerabilities for that. It goes to GitHub's security vulnerability database and see whether this package you are now about to update, does it have any issues? And it's all agentic, right? So it's about like rewarding the right behavior with a few short examples, And the only way you get to a good product, I think, is through usage, by just observing your users.

Now, increasingly, tools that get developed rely on models, the LLMs, and models have a way of at times hallucinating. How do you then take that into context? Is it through this process that we you're talking about when you're working with software that actually is built on LLM models themselves?

Right. So LLMs are, in our opinion, like more like infrastructure. It's more like serverless. Like everyone has same access to the same models, right? The difference is really the context, the entire machinery, the workflow that you build around them, right? And it's not just one model that we use, right? So it's like multiple steps. There's a lot of summarization happening, context gathering, even that's done through agents, right? And then you're using some of the reasoning models in the end, which will do a first pass of the review. Then there's another agentic steps after that. There's code generation steps, right? So it's kind of like a combination of multiple models that go into building an AI-native kind of an application.

How do you think about defining what models to use?

That's a tough question. A really, really tough question, to be honest, right? I kind of look at Twitter all day as well, like, I mean, to find like which use case would closely match mine because Each use case is so different. So when someone says this is the best coding model, but interestingly, we are not a coding system, we are a code review system, right? So generating code is a very different skillset for AI than reviewing code, which needs more reasoning.

Hmm.

For example, like companies like Cursor will use smaller, faster models. They acquired a company called Super Maven, for example, right? Because a lot of the times what you're doing is like sub-second latency tab completion, like you are showing the suggestions. as you type the code.

Right.

So you're going to use smaller, faster models with some vector search for context, right?

Yeah.

On the other hand, code review is like a PhD-like problem, right? You have to really understand the logic, the control flow, the branching, right? So over there, latency does not matter. Like for us, like we are not like a real-time use case. As you're typing, you can take several minutes for a single model call to happen, like in 1 or 2 minutes, right? For some of these O1 models are slow, right? And that's where you get very different kind of inferences, right? And the other part is like we don't believe in the external benchmarks. In fact, when we were like early days, we didn't have a lot of our own evals. We used to rely on LM-SYS and all these other evals to see which models are good, but they almost always backfired when we put them in production. So over time we built our own evals to gauge which models are truly working for us for this specific use case and which ones are not.

So you must have models that you find effective, others that you've decided to use, and maybe they're not so effective. So it's a constant review process.

It is. Like, and there's so much variety out there. Like, it's not like early days of just GPT-4. Now we see more than one organization, be it open source or closed models, right? They have reached GPT-4 levels. I mean, that was like a while back. Now we are talking about Reasoning models, right? And one after another.

Yeah, reasoning models are interesting, right? They're like, maybe it's not right the first time, but it'll come back and then say, oh, I didn't take this into consideration.

Yeah.

So it goes through that reasoning.

Right. And we're still in the early days of reasoning, right? Because these models are also improving exponentially fast with the reinforcement learning data they're getting.

So you're constantly kind of tuning the models, you're looking at the new models. Then that then also reflects in on the contextual technologies that you use too. So you have to take some approaches to think about what contextual technologies you actually apply to these problems.

That's right. I mean—

So RAG or vector approaches?

Right. To some extent, yes. We have seen that reasoning models really go haywire if you push a lot of context to them. In fact, if you're going to do RAG blindly with reasoning models, just use a top-K without any re-ranking or filtering, then these models tend to Yeah. Right? With additional context is not always helpful, right? So even like people had to redo their approach to RAG when the reasoning models came out. So it's very interesting. Like each time a new model comes out, it brings in new capabilities, but it also seems to regress on the things that seemingly used to work in the past. Even the instruction following changes. Like one of these scenarios was when Sonnet-37 came out and people were using it in coding for Internet Cursor and Windsurf. What they found is like the model was just too aggressive with the work it was doing. You asked it to do one thing, then it just goes and decides to do 10 other unwanted things, right? So we have seen that kind of challenges. And even like the cutting-edge companies in this space are still figuring it out. And despite having a lot of the evals and so on, like only in the wild that you realize, only with the usage is where you realize the gaps and the pros and cons and strengths and weaknesses of these models, right?

So you're starting to see the actual eval, You kind of get a firsthand look at the evolution of these contextual technologies in the market right now. You're basically on the cutting edge of like using contextual technologies, so you're developing your own. I'm curious on what you're seeing in the evolution of, of those contextual technologies. Are there trends that you're starting to see?

Yeah, this is very interesting. Right now we are kind of in the very early days of AI and a lot of companies are trying to build tools and platforms, including Google, right? It's early. But one of the things we have seen is that the right abstraction will take a while to emerge. Like, it's the same example with Kubernetes. Like, Kubernetes took a while to emerge as an abstraction, right? Even though we have been building web technologies and service-oriented architectures for a while. And something like that is going to happen to AI. Like, something like Kubernetes for AI will emerge, but it's too early. I mean, when I look at CodeRabbit and look at all these platforms out there, We don't feel like picking them up because our use case is so custom and we are customized to an extent that a lot of these frameworks will actually kind of box that into their model, right? We have very different requirements, right? So that's what we are seeing. Like some of these things are like maturing, like the RAG technologies are maturing over there. Like we have a little bit more stability in the retrieval process, but there is not much stability when it comes to building agents or reliable agents is a whole different story, right?

Reliable agents. Yeah.

And interesting with agents thing is like, The errors compound. So if you are hooking up multiple agents, your first level of agent, let's say 5% hallucination or errors, the next step is going to see 10% because each step you go deeper in the chain, the worse these errors become because they all add up and compound. That's the biggest challenge.

Oh wow. That's a whole new dimension to agents, isn't it?

Oh yeah. Yeah, it is a big mess. Stabilizing one prompt is a challenge, but once you chain up things in, then it's It's a whole different cascading problem, right?

So you almost have to reverse engineer the prompts.

Yes, we have to. For the chains. Yes, yes, you have to. Like, it's like, again, a different kind of a cascading failure. Like, you have to trace down where exactly things went off track in the chain, right?

And it could do things that you really don't want it to do. Yeah. This has been really great. Thank you so much for taking the time to talk. I really have enjoyed it.

And thanks for having me here. I really enjoyed the conversation. Thank you.

Thanks for listening. If you like the show, please rate and review us on Apple Podcasts, Spotify, or wherever you get your podcasts. That's one of the best ways you can help us grow this community, and we really appreciate your feedback. You can find the full video version of this episode on YouTube. Search for The New Stack and don't forget to subscribe so you never miss any new videos. Thanks for joining us and see you next time.

Bye.

See you soon.