Code Context is King: Augment’s AI Assistant for Professional Software Engineers, with Guy Gur-Ari

Hello, and welcome back to The Cognitive Revolution. Today, my guest is Guy Gerari, co-founder and chief scientist at Augment, a company using the full range of AI strategies from autocomplete to RAG to chatbots to autonomous coding agents to transform the practice of software engineering in large enterprise codebases. While our first episodes in the Software Supernova series looked at vibe coding platforms that allow anyone to prompt their way from zero to a proof of concept or basic app, Augment, which was founded in 2022, back when OpenAI's Codex models and early autocomplete tools were still mostly just foreshadowing a very different way to code, is tackling a harder but potentially more economically transformative challenge. How do you 10x productivity for professional engineers who bring their considerable human expertise to bear on vast, messy legacy codebases, which often have millions of lines of code spread across multiple projects, that can vary in age, coding style, and underlying technical infrastructure. Unlike personal projects where one can often simply copy an entire codebase into Gemini's context window, as Guy explains, the enterprise challenge requires serious technical firepower at all levels of the stack. And so Augment has spent the last 3 years deeply exploring multiple different approaches to code understanding and has ultimately developed a sophisticated retrieval-heavy approach from the ground up. Their RAG stack includes a custom-built vector database capable of real-time updates, proprietary retrieval models designed specifically for large code bases, code search that fires on every single keystroke for every single user, custom code generation models trained with a technique they call reinforcement learning from developer behaviors, and multiple different product paradigms for delivering code to users. All of which is intensively optimized for both accuracy and speed and available across a number of the most popular development environments. The results are quite impressive. As you'll hear, Guy reports that he personally hasn't written a line of code in months. These days, the coding agent, which I had the chance to use in preview and which will be released to the public very soon, handles all of that, leaving Guy to focus on higher-level issues, including how he and the team can continue to improve the agent so that it can eventually run for extended periods, take on larger projects, and even go beyond explicit user instructions to infer and address unstated needs. The economics of the business are fascinating too. Augment's pricing is pretty conventional today with $30 and $60 a month plans, but Guy was quite candid about the fact that some power users already cost them a whole lot more than that to serve. And especially as agentic workflows consume more and more compute, pricing in the AI space in general is very much a live question. It helps, of course, to design pricing that aligns company and customer interests, but it's less clear how best to do that considering that enterprise customers also value stable pricing and predictable costs. The good news for Augment is that having raised some $250 million in investment capital, they do have some time and financial cushion There is a ton of technical depth in this episode, but arguably the most valuable part is Guy's super practical, down-to-earth advice for AI builders. While he and the Augment team have repeatedly invented new technology to solve hard problems, he recommends starting new projects simply by creating small evaluation datasets of just 10 to 20 high-quality hand-labeled examples that you understand deeply and can quickly test new solutions against. And then optimizing for the speed of iteration by pursuing the simplest available strategies first, and then exhausting what's available in the market before building custom solutions in-house. All advice that, as regular listeners will know, I wholeheartedly endorse. Toward the end, I asked Guy if Augment is currently hiring junior engineers, and more broadly, what advice he has for today's early career engineers and CS students. His answer? I think you'll agree, reflects the current moment in the software industry, a sense of excitement and opportunity for the foreseeable future, but also a recognition that nobody can see the future more than 2 to 3 years out. As always, if you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends or write a review. And we always welcome your feedback and suggestions, either via our website, cognitiverevolution.ai, or by DMing me anywhere you like.

Thanks, Nathan.

Now, I hope you enjoy this deep dive into the hard tech powering AI coding assistants for enterprise software engineers with Guy Gerari, co-founder and chief scientist at Augment.

Guy Gerari, co-founder and chief scientist at Augment. Welcome to The Cognitive Revolution.

Great to be here. Thanks for having me on.

My pleasure.

I'm excited about this.

So we've been doing a little series that I'm calling the Software Supernova, which is, you know, just a nod to how much the software industry is changing.

Mm-hmm.

And we're coming at that from a bunch of different angles to try to understand it as deeply as possible. I think you're going to provide a really differentiated and interesting angle because a lot of the stuff that we've looked at previously has been kind of people, you know, want to create an app out of nothing. And, you know, there's a growing number of products out there now that actually can take you quite far if you show up, you know, with just an idea and you want to go from kind of prompt to app in seconds, you know, is often the promise. But you guys are coming at the software industry from basically the other end, which is targeting larger organizations with big code bases and things in production, long-lived projects. And so I think that it'll be a really interesting compare and contrast to understand the different challenges that poses and the different solutions that you're bringing to the market. So maybe for starters, give me a little just kind of introduction to the company and that core challenge. I think Folks that follow this feed are like paying attention to AI. I'm not actually sure how many have been in larger software organizations and would be familiar with the, you know, the particular challenges that those organizations face.

Yeah, for sure. So Augment was founded with that vision of bringing AI to bear on real software engineering challenges that show up when you're working on a large team, when you're working on a large existing codebase, because those are the challenges that the vast majority of professional developers face day to day. Definitely the larger the organization, the larger the codebase, the less zero-to-one projects people do and the more ongoing maintenance, still feature development, product development, but all has to work in the context of a large codebase. And our premise was we could see that AI technology was crossing the threshold of becoming useful. So back when Augment was founded, we had Autocomplete as a product that was out there, but ChatGPT still did not exist. But we could see how these models were getting rapidly better, and we felt that AI could play a big role. And we also felt that as a startup, if we went after these hard problems of allowing software developers to be productive in their codebases, we could differentiate because it requires a lot of context understanding. And I guess that's something we'll dig into more.

Yeah, there's a lot of dimensions to the problem. Do you want to give me a little sense of like, How big a typical Augment customer is? I mean, you could measure that in employees or, you know, number of repos or lines of code. But, you know, yeah, just how big of organizations are you guys targeting?

Yeah, so we typically target organizations that have hundreds of developers. We do stretch higher. So we have some customers who have thousands of developers. In terms of how many repos, that really varies from customer to customer. So some customers use monorepos like we do internally, and some customers use many different repositories, maybe one per microservice. So that varies a lot. In terms of lines of code, I think it starts probably with millions of lines and then it goes up from there.

So that highlights obviously one immediate challenge right off the bat, which is, you know, when I do my own little projects, my default workflow, unless I'm, you know, testing something else or whatever, is I'll usually have the AI write a little script to print the entire codebase to a single file, and then for a while at least, I can just copy that entire file, put it into the context window, and ask the AI for help. So I'll take it to O1 Pro, maybe to do some planning, or Claude, or now I've got Gemini 2.5, which can take me farther, but it still maxes out at a million tokens, which is obviously not going to handle the whole codebase. What do you feel is kind of not done well or sort of missed by maybe take Copilot if we want to pick on one, but you could say other offerings in the market that are, and maybe you could kind of cash that out to what are the frustration points or the places where you see developers just kind of not getting the value that Augment can in fact deliver? Because, I mean, we've all seen these sort of one-shot examples and, oh my God, it wrote this function for me and all that kind of stuff. But where does the conventional approach break down in practical terms?

Right. So from what we've seen, again, if we're operating inside a codebase that doesn't fit into the prompt, and today, even if we have a million tokens of context length, the ratio is roughly 10 to 1. So 10 tokens per line of code. And so still that only gets us to 100,000 lines of code, which in industry is still considered a small project. And there are other downsides as the project grows to actually putting all of that in the context that we can talk about separately. But I'd say the problem there is when you're working in a large codebase, you really have to keep in mind not just the work that you're doing as a developer, what you're focused on, but also the context. And that could be very obvious things like, I need to call a few APIs, I need to call them correctly, and I need to put in the right parameters in there.

And I—

I also need to call them in a way that respects the conventions. Maybe there are multiple ways to call them. Maybe there are multiple ways to achieve the task that I'm trying to achieve. And we want to be respectful of the conventions that are in the codebase. These are all things that, if you're a developer inside an organization, if you've worked there for a while, you're already familiar with the right way to do it. But when you ask a model to do it, an AI model, and you don't provide it with all of that context, it's going to struggle. Basically, it's going to give you bad predictions. Whether it's a bad completion or a bad chat answer. And so since at Augment we've prioritized context from the beginning, we have full codebase understanding built in by default to every feature. And so if you're getting a completion, it's going to take the context into account, whether it means looking up the function you're calling or looking up other examples of its usage, things like that. If you're asking chat the question like, where's this function that I used 6 months ago and I can't remember what it was, it's gonna search through your whole repository. And with agents, we see that this actually matters even more because when you're trying to get these models to achieve more and more complicated tasks in a codebase, the context becomes ever more important because simply because there's less supervision from the developer as the agent is working.

Yeah. Context, as Tyler Cowen says, is that which is scarce. He said that before we got into the LLM era, but it feels like it's 10 times more applicable to the LLM agents that we're all trying to figure out how to make work for us than it ever was for the humans. So I guess I'd love to just dig into sort of how you're making this work, because, I mean, one of the things that has really stood out to me, I guess a couple of things have really stood out to me as I've studied the company and used the product a little bit. One is the blog is outstanding. There's a lot of technical information shared on the blog, and that's really an excellent resource for people to get a good sense of what you're doing. And I guess a theme throughout many of those blog posts is just really pushing hard on bringing a lot of resources to bear for an individual user. So one way in which that manifests, I understand, is, if I'm not mistaken, literally every keystroke that I make fires off a thing to the server, which then begins to search the codebase to try to figure out what am I doing right now, where am I, and assemble the useful context. So maybe let's get into the sort of context management. You could take this in many different directions because I know there's a lot to it, but maybe describe from the second that I sort of open the app, and we can also talk a little bit about it's an extension of VS Code, not a fork. And there's a whole sort of debate going on there as to what's the right way to go to market. So you can take that one even first if you want. But then when I open the thing and I like, okay, here's my repository and I'm new, Like what's happening behind the scenes as it's indexing and, you know, getting me ready to like put me and the app together in a position to like really use a lot of compute at runtime. Right.

So we've actually, in the beginning, we explored several different approaches to codebase understanding. I think the approach we landed on was the third one that we tried. And each one of these was a multi-month research project to try to figure out, could we make it work? What we landed on at the end could be described as, as RAG. And so what happens behind the scene is we upload the code, we have our own custom trained retriever models that we train for the purpose of codebase understanding, and then we index code using these models. That's what happens when you open Augment and it says indexing your codebase. Once that's done, yes, on every keystroke and on every chat request, we send a request to the model. And part of processing that request is figuring out which parts of the codebase are most relevant to show to the model. So it that can make the best possible prediction for the user. And there is quite a bit of optimization then that goes— the speed optimization that goes into making all of that fast. Because it's one thing to index a large codebase in the background, but then it's a whole different story to say, okay, this completion request needs to finish within, let's say, of the order of 300 milliseconds. And that needs to account for both retrieving everything that's relevant from the codebase and actually doing the language model call request.

Yeah.

To generate the completion. And so we prioritize both quality so that the retriever is good and end-to-end, it actually feels like it understands your codebase and also speed, because to us, speed is a super important feature basically of the product.

Yeah. Can you, you know, you can obviously calibrate exactly how much you wanna share about the details here, although I recently did an episode also with Andrew Lee of Shortwave and they have a pretty similar approach where you sign up and the first thing they do is ingest your entire Gmail history, which can be a lot, right? And then that goes into their database. And he's like a database guru. And he told me, and I don't know if you'd feel the same way, he's like, well, yeah, we can pretty much tell all our secrets because by the time anyone figures out what we've done and tries to recreate it, we'll have a whole new generation. So I'm not sure if you feel quite as confident on that. dimension and would be willing to share all the current way things work. But to the degree that you can, long preface, I'd love to understand a little bit better, how are you chunking code? Because I think people have broadly kind of come to frustration with RAG where, and I think there's a number of different reasons for this, but sometimes you can kind of fail at every step, right? So how are you chunking? When you get a hit on a chunk, are you then expanding out to make sure you have the surrounding context that's needed. So it's not just that one function out of a broader class loaded in isolation. There's context management and then there's effective context management, I guess, is what I'm really trying to get at. So how do you make it not just fast, but actually good so it has the right information that it needs?

Right. So I probably can't be as open because I do believe that there is quite a bit of secret sauce in what we do. I mean, it is true. Getting RAG to work well is very challenging. And as far as I— in my experience so far, getting it to work well on code is even more challenging than in other domains. So just to give an example of why that is, let's say I'm starting to type something. And I'm starting to type a piece of code. And let's say there's enough context there to understand kind of what it is that I'm trying to do. Maybe there's a comment or something, although often there's not even that. And my cursor is sitting there and I'm trying to— we need to get a prediction out of the model and we need to know what pieces of code are relevant to make that completion. This is a very different situation from chat or question answering system where the user is asked to provide the context for the request, right? You start with an instruction or you start with a question. You have basically a lot of context for understanding what it is that you're going to be looking for in your knowledge base. With code, at least with chat, you kind of have that, but with completions, it's more passive. We kind of are trying to both infer what it is that the developer is trying to do and then try to figure out which code it is. So just to give an example, let's say that we figured out that we need to call a function, or the model figured out that it needs to call a function. Then the question becomes, okay, what pieces of code are most relevant to help the model make that function call correct? We could pull up the function signature. We could pull up example usages of that function. We could pull up other pieces of code that maybe serve as counterexamples. Another thing about codebases is they evolve over time, and we kind of see a snapshot of the codebase. And so, if we're pulling up examples, those examples could be new, or those examples could be obsolete, and they're just left around in the codebase, and actually, the developer doesn't want to call them that way. So it's an extremely challenging problem. What I can say is that we use a mix of different techniques. We use RAG, we use some amount of static analysis on code. There are multiple models at play to provide the best possible context to the model. And we also let the user steer often because these systems are not perfect. And so we need a way for the user, especially in chat, to say, okay, I'm actually pointing at this directory or I'm pointing at this file. And this can also kind of indirectly help steer the retriever. Yeah, there are multiple things at play. I'd say on chunking, yeah, there's definitely better and worse ways that you can do it. And it's true that, yeah, code has more structure that you can hang on to. What I can say is, this is more— yeah, I can say that improvements in chunking are more to solve problems that are in the tail. Maybe I can say it like that. If you have strong retrievers and strong models, chunking shouldn't be a blocker. Yeah, I think I can say that.

Hey, we'll continue our interview in a moment after a word from our sponsors. So yeah, one thing that jumps out to me there is just how you started with the sort of assumption that the user is like typing code into an IDE in the traditional way. And this maybe can also tie back to like, the go-to-market as an extension of VS Code as opposed to a fork. I'm so AI-pilled myself and just kind of always trying to do 2 things at once. Usually I'm trying to accomplish some goal in some project, but then also learn about the latest AI capabilities or use Augment or whatever, right? So I'm always kind of looking for these 2-for-ones. And I think that probably puts me, I'm realizing, in maybe a very different pattern of behavior than what you typically see. So as I've been using it over the last few days, I've done it entirely through the sort of chat panel and I basically don't really ever, almost never, actually get in and start typing functions myself anymore. Where are people on that today generally, though? What is the sort of balance of approaches that you're seeing from, and by the way, sort of a mid programmer, which is maybe why I'm so drawn to the chat experience. But for the pros, what's the balance between how many are kind of working the old traditional way of being file by file and then getting this assistance sort of proactively served up to them versus those that are actually saying, okay, I want to interact with an AI and have it help me, but I'm going to give it an assignment in a sort of chatter or agent type of paradigm?

Yeah, there's definitely a distribution. So I think when we were talking about completions and chat, we did notice that there do seem to be 2 camps of developers. And of course, there's a lot of overlap. Like, I doubt there are many people who only use completions or only use chat, but there are certainly developers with a preference to being a lot closer to the code, I would say. Who don't even use chat much, but really love completions. And now also next edit. Next edit is kind of a way to— okay, you get completions, they might be away from your cursor, they can delete code and edit code, not just add code. But it fits in very nicely with the workflow of developers who kind of want to keep their focus on the code. And then we see a lot of developers who really only use chat. I mean, that is fairly common. The thing that's changing now is as we're building agent mode, you can take another step away from the code and really let the model edit multiple files, run your tests, and you're kind of taking another step away and supervising everything. And then you can dig into the code when needed. This is something that we've seen when, when you work on a large code base, you, we pretty often have to go back to looking at the code and making some changes yourself. That's pretty frequent. Which is quite different from the 0-to-1 experience that I think we talked about before. So I would say for large codebases, as far as I can tell, most developers are comfortable being in chat a lot of the time. And yes, using completions, next edit, but using chat a lot of the time. The switch or the move to full autonomous agentic flow will take longer. I think agents and models will need to improve before that becomes kind of the default mode for enterprise developers. Let's say it will take longer, but I feel like that's the direction we're going in.

Yeah, certainly. And is the, you know, this is, it can get into, you know, sort of almost ideological territory very quickly, but I'm sure for you it's much more a practical question around extension of VS Code or do a fork, is that a matter of just kind of meeting developers where they're comfortable and like not asking them to change too much? Or are there other, you know, big decision drivers there that have you in the extension paradigm?

Yes, I think it starts from meeting developers where they are. And so we have a VS Code extension, we also have JetBrains extension, and we have Vim support. This is really about, we don't want to force developers to change how they work. I think with the forks, these are all VS Code forks because VS Code is open source. And so you can say that if you switch from VS Code to a fork, you're not changing your workflow that much. But if you're asking a JetBrains developer to switch to a VS Code fork, that's a pretty substantial change to their workflow. So that's one. I would say there are also other considerations with the fork, which is that doing a fork means you need to keep track with updates, especially security patches, which then becomes extra maintenance work that you have to do. And then especially if you're selling to enterprise, these security considerations can matter. Now, the downside of not having a fork is that there are certain UI things that are harder to do, or sometimes impossible to do. Although I have to say that with the VS Code API, we've been able to do a lot within VS Code. I don't think this has been a, like, a very Substantial limitation. Sometimes we've had to work harder because we can't just go and change the VS Code code. This is another place where I suspect that the more we move to agentic flows, the less we have to do kind of inside the text editor. And once you're building an agent inside VS Code, you have a lot of freedom of what to do because you can open panels, you can put web views in there, and you have full control over what's happening. So my sense is that this distinction is going to become probably less important over time. But, you know, I can't promise we won't do a fork at some point. There's certainly a trade-off there.

Yeah, that's interesting. I mean, the point about just security and being able to kind of piggyback on, you know, all the hard work that Microsoft has already done to establish trust definitely makes a lot of sense. I've had enough experience with the security review processes at enterprise customers, and not nearly as much as you've had, but I've had enough to know that it's not where I want to be spending my time. And to the degree that you can shorten that process, it certainly has a lot of appeal.

Yeah, exactly.

So going back to the just retrieval, and again, you can kind of calibrate your answers however you want, but practical guidance for other people building their own RAG apps, do you have a favorite vector database?

So we actually built our own vector database. I can explain why we did that. There was nothing out there that we found at the time that addressed all of our requirements. So, you know, what do we want the user experience to be, right? We want the user to feel like the model understands their whole codebase. And we want it, we want it to feel like it understands the current state of the codebase, right? So if I just wrote a function in a file or had ChatGPT write it for me, and now I ask ChatGPT, let's say, okay, implement the tests, or I go to a test file, and I start typing a completion, or I start typing a test, we want the model to understand that this is something that I recently did and have that all kind of indexed and available. And so that means giving every developer, or giving the model a real-time view of every developer's codebase. So it has to be real-time or feel like real-time. And it also has to be different for every developer, because if I'm a developer on a team, I work on my feature branch, you work on your feature branch, we cannot have those things mix. That's also a security requirement. So that means that in terms of a vector database, you need something that allows almost real-time updates to the index, which is already a significant requirement from a vector database. And it also needs to be able to have queries based on different views, right? I have a slightly different set of files that I'm retrieving from than you. But we still want to deduplicate, we still want to have like one database that captures our repository and not like deduplicate that for every user on a team. We did not know of a product that did all of that. And there's a technical reason for it. So typically, the way vector databases work, when you query, it's pretty expensive to do a full query every time. And so you apply some kind of statistical algorithm, maybe you cluster your embeddings and you search in the cluster. I mean, there are all kinds of ways to do that. But taking that kind of approach, or one of these standard approaches, means that it doesn't work well both with indexing or updated indexing, because updating the clusters can be expensive. And also views or queries based on views is hard, because if you're doing a statistical query, and you only have a subset of the files that you're retrieving from, you might miss them completely when you're doing a statistical query. And so it was certainly a difficult engineering problem to build a vector database for us. And we still keep iterating on it, especially as we have customers that have larger and larger codebases. We need to keep scaling up solutions. So there's a project ongoing right now addressing scale requirements with repositories for us. But yeah, we ended up building our own.

It's fascinating. I mean, I guess you started in 2022, is that right?

Yes.

Yeah. So I want— and it may or may not be different today. I wonder, you know, this sometimes strikes people as crazy when I float ideas like this, but that almost sounds like a product unto itself. Have you thought about that?

It's come up. The thing is, when you— it feels like When you build an AI lab and then an AI product on top of that, you run into many things that can become a product on their own. And one of the challenges is to stay focused and have a concrete vision of what we're trying to accomplish. So this comes up and, oh, this could be a product. This comes up of, oh, should we fork? And then, well, who are our users really are? Who are we catering to? And are they going to want to fork or are they going to want to prefer their IDE? There are questions like this that come up all the time, especially with something like AI, where it's a completely new technology that keeps improving rapidly, and you keep kind of having to keep up with what's happening and make the right bets. Short story, yes, I think this could be a product, but we're trying to stay focused on building the best AI assistant we can for developers.

Hey, we'll continue our interview in a moment after a word from our sponsors. Are there any, like, I'm guessing you probably haven't kept up with the evolution of other vector databases, but for people who are trying to pick one, because so many people right now are at the stage of either we're embarking on a sort of RAG app for our business, probably for internal use, or maybe we made one and it's not quite working well enough and we want to take some next level step with it. Are there any sort of general guidelines that you would give people for how to make this part of the system work? Like, one that I have in mind, it sounds like you kind of have a version of it, is just like, I personally would, I think, almost always insist on some sort of hybrid structured query plus vector, as opposed to At the beginning of this RAG wave, people were just doing pure vector search, and that seemed to be kind of a mess. So having some ability to do classic SQL-style WHERE clause along with the vector similarity, whatever, seems important to me. But I wonder, what would your sort of lessons or guidance for the masses be based on all this experience?

Yeah. I would treat this as a research problem and start with— I mean, unless there are pretty clear engineering requirements that would preclude— we have, I'd say, pretty special requirements because it all has to be low latency and so on. I don't think most RAG implementations need all of that. And so I would probably start with some off-the-shelf vector database, and I would focus more on the quality. And I think for the quality, one thing that's pretty important is to have an evaluation dataset that you trust. It doesn't have to be a huge dataset. You can start with even 10 to 20 samples labeled by hand. That's how we start most projects. Actually, most research projects will start with collecting 10 to 20 samples labeled by hand and then start with some baseline, take an off-the-shelf retriever, whatever is easiest to use. Run it on your evaluation and get a baseline of how are we doing? Are we solving 20% of samples? Are we solving 80% of samples? Probably it's going to be somewhere in between. Is that good enough? And start iterating from there and hill climbing on your evaluation dataset. And when the evaluation gets saturated, so basically when you've managed to solve it, expand it, add more samples, make them more diverse, make them harder. I think this is coming up with good evaluations and being diligent about running those evaluations is, I would say, in some sense, it's one of the hardest things to do in research, not because the work is so hard, but because it can be pretty tedious. But this is the way to get to good results. And so starting with that, things become straightforward in terms of, oh, should we just do vector? Should we do structured queries? Well, let's try it. Everything becomes an experiment. Let's try it on the eval set. And the eval set will tell us because we're basically reducing the problem to hill climbing on an eval set. That is the ideal situation. I will say yes, certainly doing pure vector Let's say it like this. Real-world retrieval systems are almost never a single thing. It's almost never, oh, I'll just do embeddings, and I'll work on the embeddings really hard, and I'll get the best embeddings, and they will solve the problem. That almost never happens. It's usually a mix of different techniques. So vectors, structured, could be other, any other signal that you can bring to bear on problem. And the models today are good enough that you can actually throw a lot at them in the context, and they will kind of deal with it. And so in some sense, recall becomes more important than precision. You want to make sure that the right chunks are in there in the context. That's really today with modern models, that is the thing to prioritize. Now, I described like the ideal situation where you can kind of come up with a dataset that you trust, and all you're doing is hill climbing on that dataset. It's very important to start like that. But at some point, necessarily, your evaluation dataset is going to diverge from optimal user experience. And again, I've never seen it— this has happened in every one of the projects, I think, that we've done. It's really hard to capture user experience and map that to one number, not just because there are multiple axes, but also because we don't really know how users use AI products. There's a whole distribution of what they put into the prompt box. There's a whole distribution of what they expect to get out. You can't really boil it down to a number. And so once you have something, dogfooding is crucial for understanding where you are. And then once you have users, user feedback is crucial. You have to take all these things into account. So I would recommend starting with an eval set, but then understanding that you also need these other sources of feedback to iterate. So that, I think, is a summary, kind of like a quick summary of best practices for how to go about this. Once you're able to reduce all these questions, basically the place you want to get to is, can you reduce all these questions of what to do, turn them from philosophical questions to experimental questions that you go test? Then that's when you can iterate and really move fast.

Yeah, that's great. It is striking to me, and for whatever reason, I find myself doing more projects where there isn't quite a ground truth that's so easy to hill climb on. Like with my company Waymark, we do video creation for small businesses, and there's not like a single answer that's like, what is the right or best script or voiceover script or whatever to, or selection of images? For this particular small business. There's definitely better and worse, and sometimes it's very obvious, and other times it's the subject of disagreement. We've definitely had plenty of cases where we ask 2 different people and they see one is better than the other. It's not unanimous in most cases, but it is definitely an enviable— from that perspective of having these sort of almost irreducibly vibey sort of tasks, the idea of being able to just climb a hill is quite attractive. But in both cases, I do think it's really important for people to keep in mind you can start and you should start with a pretty modest sized dataset. I have a whole presentation that's much more about the social side than the technical side of getting your team on the same page on what 10 instances of a task that are really well done look like. And it's amazing to me how often That ends up becoming the stumbling point. I think it is often because of what you said. It's just kind of tedious. And there's like, you know, they all also like don't have any chain of thought, you know, which is sometimes really helpful if you want to do a, you know, supervised whatever. I'm on the verge of the soapbox, but yeah, 10 examples, people, will take you far. If it's objective, great. Even if it's just like a vibe task and you're just demonstrating what a job well done looks like, 10 examples, you know, it's the first place to get. And then from there, you know, the world can open up like Yeah, just to add to that, 100%.

So these few examples, the big advantage of having so few before you go to like hundreds, right, if you can, is that you can, you become very familiar with them, you can hold them all in your head. And so the labels, the ground truth is less important. I'd say you can, you can still hill climb on 10 samples, even if the evaluation procedure is completely manual. So I just trained a new model or I have a new RAG setup. I will run it through the 10 samples. I will run the before model and the after model and I will compare them by hand. I don't have to have a— I can boil it down to a number, but I can also go based on vibe. So I agree with you that the real minimal thing to start with is the 10 samples. The number can— yeah, the number can come later. I totally agree with that.

One other thing that you said that I thought was worth just reemphasising too is the key thing is make sure the model has what it needs. Worry less about other considerations like distracting it with wrong information. And this is, of course, evolving quickly because that used to be, I think, a much bigger problem not too long ago. And as you said, modern models, that word modern is important. Everybody should be using modern models. But Our expectations aren't necessarily always keeping up with what the latest models can do. One way I kind of generalize that for people is just like, turn your hyperparameters up in general. You have a choice typically in a lot of these RAG type setups of how many chunks am I going to take? I'm going to take the top N chunks, or if I'm going to expand out from a chunk, how much should I expand out? And I'm always kind of telling people, from what I see developers doing, they're leaving all those sort of settings too low. The right thing to do is usually turn them up. Yes, that might make it slightly slower. It will make it a little more expensive. But it's always like, at least I don't know if I can think of any exception where turning those things up didn't more than pay for itself, even with those marginal cost increases, in the sense of the time savings that you get to getting to something that's like working better. So I don't know if you have any exceptions you would put on the turn your hyperparameters up rule of thumb, or is that also— I mean, from the blog, it does seem like you guys are definitely like, how can we sort of jam and use all these things to roughly the maximum? But yeah, interested in your take on any nuances you would add to my simple rule.

No, I fully subscribe to that rule. You just have to be aware of the trade-off between latency, cost, and quality. It's really as simple as that. If you're okay with the extra latency and cost in RAG context, it's just better to show more because these days the models have been trained to deal with it. It didn't used to be the case, like definitely not 2 years ago, maybe not even 1 year ago. I'm not sure. But roughly around that time is when models got the RAG training to be able to deal with A lot of distracting information. And so that scales really well. And I expect we'll just continue scaling because the attention mechanism in transformers is basically built to do that. It's basically built to sift through all the noise and focus on the relevant parts. And so with sufficient training, it makes sense that it will work. I think I've also seen this bias toward putting less, less in there, especially if you're coming from a background of using models the way they were 2 years ago or Or before there was, you had to be a lot more careful with your tokens, but that has changed. I think the place where, so if we're focused on, if we're talking about RAG, then yeah, I think that's just the right answer. And also certainly way easier than on the research because making, improving the recall. So that is improving the ability of the model to find the right chunk within the first like 50 or 100 or something like that. Is doable. Improving the recall, if you have fewer chunks, like if it needs to land in like the top 10 or something, or top 5, that task becomes exponentially harder, basically, the less context you have. So if you can give it the room to do it, then the research task becomes much easier. I can say the place where adding more context doesn't seem to scale yet is with instructions. So giving the model tons and tons of instructions that you expect it to follow, in my experience so far, this actually doesn't scale that well and it will start ignoring instructions if there's too much in there, skipping steps you asked it to do, things like that. But that's not a RAG problem. That's just a different kind of prompt scaling problem where models are not yet good enough.

Yeah.

Okay. That's a good point. How about you mentioned Training, you've trained your own retriever models. This is something that I think, I don't want to bias your answer too much, but my sense is that a lot of software engineers, AI engineers are sort of attracted to the notion of, well, and of course they're not going to pre-train from scratch, but we'll grab some off-the-shelf thing and we'll customize it for our own purposes. I wonder what guidance you would give people on when that is in fact a good idea. I once made a meme, the bell curve meme of what's genius and what's dumb and what's in the middle. And my thing was on the extremes is just like, just use OpenAI embeddings. And in the middle was, well, we'll do this complicated thing, we'll fine tune our own, blah, blah, blah, blah, blah. It's worth it for some. You're in that situation where you're have a lot of resources and a very ambitious project, but where do you think it starts to become worth it to take on that sort of challenge versus just using the best off-the-shelf thing you can find?

Yeah, that is a very task-dependent question. Yeah, I think, I mean, anytime you do research, I think there is just a general human tendency to reach for complicated solutions too quickly. My recommendation would be to try hard to bias towards simplicity. Like, really, the simpler the better, and rely on your evaluation, either like vibes-based evaluation or numerical evaluation to guide you. Yeah, I'd say, again, it's hard to give general answers, right? Because the answers are going to be task-specific. So I'd rely more on establishing a solid process for finding the right answers for your particular use cases. That starts with an evaluation set that you're comfortable with and that you mostly trust and that you can run through different iterations of your model or your system to test. The other thing that I think is important to optimize for is iteration time. Because the faster you can run experiments, the more likely it is you'll find something that's good enough or something that's better than what you currently have. So experimental iteration time is something that's very much worth thinking about. Taking experimental time down from hours to minutes can have a lot of impact on not just how fast you get to a solution, but actually, do you even get to a solution? Because cranking through 100 experiments versus cranking through 10 experiments, the chances of you finding the right thing in those 100 experiments is just much higher. It's kind of like the RAG problem. If you, like, what's your chance of finding the right chunk in the top 100 versus top 10? It's just much higher. It's also like that with experiments. So if you can afford to run 100 experiments, then yeah, you're going to try 100 different simple things. Maybe you're going to try SQL-style queries. Maybe you're going to try 5 different open-source models and find that, like, maybe you'll try the OpenAI embeddings plus other things, and you'll find that one of them, for some reason that was really hard to predict, actually works better for your use case. So I'd prioritize experimental iteration time and being able to actually trust the result of an experiment with an eval set to tell you the answer. And once you've tried some simple things and nothing seems to work, like, I think another thing that process gives you is a kind of feel For, it looks like nothing out there is really doing what I want. Maybe I should start thinking about fine-tuning an open-source model. Or, oh, okay, this is like, it's not exactly there, but it's kind of close. It's not that far. Probably if I keep going this way, I will be able to make it good enough without fine-tuning. That's the sort of information you get by doing a lot of experiments. So my suggestion would be to do that and then let the experiments kind of tell you which way you need to go and when.

Okay, cool. That's great.

There is so much that I wanted to cover, and I don't think we're going to get to all of it, so I'm going to have to start to pick and choose, and then I'll refer folks to the blog for some deeper dives on stuff we don't get to. But one I definitely want to cover is reinforcement learning from developer behaviours. Obviously, reinforcement learning on language models in general is having a moment. The floor is yours. Tell us about reinforcement learning from developer behaviours.

Yeah, so one advantage that we have as a company that both does research and builds a product for users is that we are very close to our users. We get feedback from them on Slack, on Discord, and they also send us their data. Now, for enterprise customers, we do not look at that data. Of course, everything is audited and behind access controls and so on. But we do have a free community tier that's for anyone who wants to use Augment. It could be on open source. It doesn't have to be on open source, but anyone who's comfortable with us looking at their data and also using it for improving our own models. So there's a very clear separation between those 2 things. But on our free tier, certainly we find value from that data because one of the things that's just universally challenging about building AI products is that we don't really know the— let's call it the input distribution or the task distribution. What do users want to do? How are they trying to use the product? What do they expect to get out of it? And this is through collecting this data from the free tier, we get a glimpse into that. And in fact, for coding, we get more than a glimpse. So that, that is one of the nice things about coding is that it's quite different from a chat interface, right? In a chat interface, the user asks a question, or they assign some tasks, they give an instruction, they get an answer, they can continue steering. But we don't know what the ground truth answer was. By comparison, if they're working on code in their IDE and we follow what's happening in the IDE, we eventually know what they were trying to do because this is where they actually work. And so the way this connects to reinforcement learning is that the idea with reinforcement learning is that you are not just training the model by showing it examples of what to do, You're actually showing it contrasting examples. Every sample contains input, and then it contains a better and a worse output. And the model learns from that contrast to do better over time. This is a very powerful paradigm because it means that it's not just that there are correct and incorrect answers, there are better and worse answers. And you see that in coding just like anywhere else. You know, the answer could be correct in the sense that the algorithm is correct, But it could have the wrong style. Maybe that's not what the developer prefers, or maybe the style does not align with the rest of their codebase, right? There are actually multiple axes of— in which a sample can be better or worse, and that's the kind of signal that reinforcement learning tries to capture. And so we've applied that technique initially to the completions feature. I mean, that was kind of our first reinforcement learning project where we use examples from the model and we use what we know from the user in order to improve the model and align align it better with what users expect through reinforcement learning. And we ended up with a better completion model because of that. Because I mentioned that we train retrievers, but we actually also train the generation models that we use for completion and next edit.

There's a lot that I am interested in digging into more deeply there. I guess for one thing, have you benefited from DeepSeek and other recent algorithm releases? Has that stuff— I mean, I think it's safe to say the GRPO broadly has blown a lot of people's minds. Did it blow your minds, or did you feel like you already had a pretty good read on what was working such that it wasn't such a revelation for you?

The algorithm itself, I don't think was— I'd say, yeah, I think there were known problems with existing RL algorithms. That every subsequent iteration kind of addressed. So yeah, I don't think GRPO for us was a— it was more of an iterative improvement. I think the DeepSeek work was remarkable. I mean, in the sense of, yeah, they actually implemented chain-of-thought reasoning training. It was super impressive work and very nice paper. I wish they shared more details on how they did it, but I still enjoyed reading the paper. We do benefit a lot, so not from that particular work, but I can say we don't train models from scratch. We made a bet very early on that open-source models will rapidly become better. And that was at a time where open-source models were really not good. This was long before LLaMA. I previously worked at Google, worked on some of the training of large models. And so it was clear that this is something open source could do. Because there is, at least at the level of training base models, and then also at the level of like, now we also see beyond base models, like actual instruction-tuned models, because there is at this point almost a playbook for how to do it. And so if you have the resources, and if you have the people who know the basic techniques, which are for the large part in the literature, like you can read papers and learn how to do it, then you can train very good models. So we made that bet early on. We don't train from scratch. We do a lot of post-training on models for retrieval and generation. And so we definitely benefit from open source models coming out, and we generally try to keep up with basing our models on the best available open source model that's out there. Yeah.

So this, I think this particularly caught my interest, this reinforcement learning from developer behaviours, because I've been kind of looking for something like this to emerge for a while. My sense is like, the compute requirements for the reinforcement learning aren't so crazy, and the datasets don't even have to be so huge. It seems like a lot of sort of product user bases or just communities potentially in general, if they're passionate about a certain subject or whatever, could kind of gather enough feedback or behaviours from people to power this sort of thing. I haven't seen too much of it, and I was wondering why it wasn't happening, and now it seems like it is happening. I wonder, though, if you could shed any light on where you think it's going. In particular, one might think that the fact that there's not a true, absolutely canonically right answer, as there is in a math problem with a numerical answer, would suggest that maybe this process would sort of top out at human level and might not go past human level, because how would it go past human level if it's learning from humans? The flip side of that also would be that it would seem like this approach would be very extensible to reinforcement learning from lawyer behaviours or reinforcement learning from doctor behaviours and other, basically anything where you can gather enough data that's pretty trusted, even if it's not absolute bedrock ground truth. So what do you think? Is there a top out that we should be thinking about? And is there any sort of limit on the sort of breadth of how far these sort of approaches could generalize?

Yeah. So I guess, well, a few thoughts. So first, if we think about the trends we've seen in the beginning, starting with GPT-2 and then GPT-3 and the scaling laws trend, what was the trend? It was there's a whole lot of data out there on the internet. Let's get as much of it as we can, process it properly, clean it up, filter it, because there's also a lot of garbage out there. But basically, that was the first resource that large language model training reached for. I think at this point, that data resource is more or less exhausted. And so what can we do? There are 2 things we can do, I think, roughly, or 3 things, let's say 3 things we can do. One is synthetic data. I mean, we know that we can generate more data. Out of these models to train new models. So that's certainly one approach. Another approach is pay contractors to give us the data that we need, right? And that's how most RLHF works. That's how you train something like ChatGPT, essentially, if you don't have any other data sources. And the third is user data. So if you have real user data, if you can figure out how to use data from users who are using your product to do real work, that's in some sense the holy grail, because that is the closest you're ever going to get to the actual distribution of what users are trying to do, because it is what they're actually trying to do. So there's no distribution gap in that case, if you can do it well, between the data you're training on and the data you're going to encounter in the wild at test time. Now, I think the reason we haven't seen more of that is that there aren't There aren't that many products that are amenable to that. So I think if you mention beyond coding, doctors, lawyers, what do they do where you can actually get the ground truth? If they're editing a document, you can get the ground truth just like you can from an IDE. If they're using a chat interface, it's a lot harder to get the ground truth. Maybe you can guess at the ground truth because, you know, maybe they tell you, no, no, no, that's not what I meant, do this or that. And then you But there's a lot more work that you have to do to extract the ground truth from something like that. But I do believe that as we exhaust the available information on the internet, user data is just going to become a lot more valuable and people will pay more attention to it. In terms of tapping out, well, I would just say that everything we've done so far is train models based on human data. There's nothing really new here. All the data from the internet is human-generated data. RLHF is human-generated data. We can automate some of it. So we can throw models in there and let humans supervise at a higher level. But human supervision is always there so far. In the future, if we want to break away from that, we need some other source of signal, right? We need some other source of reward for these models. That's where I think code is probably the place where it's going to come first, because the thing that's special about code is that you can execute it and you can get feedback from that. And so I can see how in the future we'll be able to do— and some of it is still happening. Like if you look at the way DeepSeek was trained, they don't say a lot on how they did it, but they do get feedback from code execution for RL purposes. It's a very natural fit. So for code specifically, I think we'll be able to do a lot more of that. For other domains, we'd have to find something else. And so, you know, if you're asking the model to write a story or a poem or an essay, how are we going to automatically assign a reward to that if we don't already have a better model that can judge what this model did, right? So that's where I can't think of a way to go beyond human capabilities there. But when you do have a ground truth, that's separate from humans, code execution, maybe for science, this could be experimental validation, things like that, then we'll be able to, yes, at some point shift away from humans and rely on these other reward signals.

Have you seen any reward hacking in your reinforcement learning from developer behaviors work?

Oh, in our reinforcement learning? No, no, I can't think of an example. No, there was nothing as spicy as that. The mistakes were, yeah, just not understanding what the user wanted was just the, I think, the most common mistake. Yeah.

Okay. Well, keep an eye out.

Yes.

We're all looking for reward hacking. We should be these days, I think.

Yeah, yeah. Yes.

So let's see, just prioritizing, kind of triaging a little bit.

Maybe a minute.

I can give shorter answers also, if that helps.

No, you're doing great. Maybe a minute on just the economics of businesses like this. So it's public, and again, you can go into as much depth as you want. It's public information that you guys have raised $250-ish million. And I looked on LinkedIn, I saw 100 employees. Perhaps not everybody's listed there, but if I were to just do, like, kind of traditional SaaS app math and take a multiplier of employees times some Bay Area salary and then try to calculate a runway, I get to a really long runway. And so I guess I'm wondering, kind of, and that's before any revenue, and it sounds like there's quite a bit of revenue. So what are you doing with the money? Are we burning a lot on training models? It sounds like if you're not pre-training, that doesn't seem like it would be Are you subsidizing users? I mean, you are subsidizing open source. So, yeah, I guess maybe to the degree you can, I'd be really interested to hear about the economics. And then a slight extension of that would be, is there a 10x more expensive version of the product that you could dream of or imagine what that might look like?

Yeah, that's a great question. So Yeah. One thing we've learned is that AI is different than— yeah, we're all used to thinking of SaaS businesses and like you're developing in your basement, you set up a Google Cloud project, you start serving users, and it's all very— it's pretty cheap, I would say, right? Your main cost is salaries and so on. But AI is more capital intensive. There is the training, but inference is also very expensive. So serving all those requests at every keystroke and then the chat requests and then now the agents, it gets expensive quickly. Yeah, there's some amount of subsidizing users like in the free plan. And I think everyone in this space is kind of trying to figure out right now the economic model because I would say, you know, on the one hand, usage is exploding and we can talk more about— I think it's going to accelerate dramatically and we can talk about why that is. Models are getting cheaper, but they're not getting cheap. It's not matching the pace at which the usage is growing in this space. And also, yeah, they're getting cheap. I mean, a given model is getting cheaper, but people always want to be on the latest model. And that's not getting cheaper as quickly. So all these factors combined, it's running an AI company, Both if you're training your own models and also if you're not, can get pretty capital intensive. So that's the short answer. Now, on a more expensive product, so if you just look at the shift, so we're launching our agent feature. The cost of agents compared to chat, for example, is it's a substantial jump because with agents, you give it one instruction and then it goes and that will probably generate 10 or more language model calls, including large calls for like editing files and like running whole commands and then parsing their outputs and doing all those things all from a single user instruction. And that's at a point where we are giving users a single agent in their ID. So it's just going from chat to one agent. On the other hand, the value is clearly there. So I can say personally, I think I have not written— I've been using our agent. I have not written a line of code in several months. I've— the agent has written a lot of code. I personally have not had to. The value is very obvious with these things, and it's super early. I expect usage of agents to explode over the next year, and with it, the cost. So if you're talking about like a large jump in cost going from chat, let's say, to agents.

Yeah.

There's going to be, I expect, a jump that's at least as high once we're able to unlock the full value out of agents. And I don't think that cost decreases in models are going to keep up with that. And so cost does become a challenge. So the whole thing is just very capital intensive. And cost is actually a major factor. Unlike traditional SaaS businesses, I think, where it's not as much of a factor.

Are you managing your own clusters? Are you doing the actual buy up all the GPUs and manage them in-house? You're not leasing or renting from some other provider?

No, we are leasing. We are leasing. Yes. We're not in the business of managing data centres. We are leasing the GPUs. Yes.

Yeah.

Gotcha. So does that translate to a higher price point at some point in the future? My rule of thumb has been, I think companies should expect to spend $1,000 a month on AI to augment their employees, no pun intended, in the not too distant future. I personally am probably halfway there just with stuff that I've signed up for. And then your prices are $30 and $60 a month. And it feels like if I'm right, you should be probably like 5 to 10x-ing those prices. But I don't know, is that where you think it goes or not?

I don't have definitive answers. To me, these are open questions. There's even a question of should it be a fixed subscription price or more of a consumption model? So, you know, we actually started our current pricing model is a somewhat consumption-based model where we sell credits. And then if you use the product for a given month, if a developer uses the product, they consume a credit, but if they don't, then they don't consume a credit, which is different from the, I think, more common seat-based pricing where you just sell seats and you pay no matter if users use it or not. So we already took a step in this direction of consumption-based pricing, which was meant to really align our interests with those of the users. Like, you use it, you pay, you don't use it, you don't pay. And I expect because of the cost, we're probably going to lean, my guess is, more heavily into that model, but I'm not sure yet. So this question of, is it going to be $1,000 a month? Maybe we'll end up there, or maybe we'll end up with a different model that's kind of more aligned with how users actually use it. What I can tell you is that there's a very wide distribution in how users use these things. There are absolutely users who will justify a $1,000 a month price point even today. And then there are users who don't, they just don't use it as much. And so I think we and everybody else are kind of trying to figure that out. Let me throw another thing in there. I think right now we're all just thinking about user-driven agents, let's call them, or interactive agents where the developer is kind of there, maybe they go get a coffee and come back. Because it takes the agent a few minutes, but it's like, you know, it's a few minutes and the developer is kind of staying up to speed with what the agent is doing. I think things are going to evolve rapidly over the next year. And it's not even clear to me if that will continue to be the dominant use case. I'm pretty sure we will have agents that run for hours or overnight or over days to accomplish tasks. I'm pretty sure we will have agents that work on non-user triggers, maybe it's API calls, or maybe it's an agent that goes and does code reviews for you and things like that, that are not just run automatically. If you're in that world, then you're not even talking about per-developer pricing exactly anymore, right? You're kind of putting intelligence into a lot of tasks that are not triggered by the user, or maybe there's a wide variance in the cost of what the user triggered. I think the pricing model is going to kind of have to adjust to that, at least in the short term until all this stuff becomes super cheap. So yeah, it's a complicated question and I don't have a good answer for it. All I can say is it's a good and complicated question that we're definitely thinking about.

Yeah, I think that it feels to me like aligning interests with users is like, it's hard to go wrong when you're generally keeping that in mind as a true north. The thing that I am always kind of allergic to is when I feel like the product is not performing as well as it could for me because I've got some fixed price, and they're trying to keep my cost to them under that price to maintain a margin. And it sounds like you're not doing that by just basically accepting the fact that you'll have some $1,000 a month cost users and kind of figuring you'll figure that all out later. But yeah, it's definitely— I want to be able to be that $1,000 a month user, even if I do have to pay for it. What frustrates me is when I can't be because I'm kind of locked into more conventional price point. So, okay, time is short. Maybe 2 more questions if we can fit them both in. One is, you guys have a blog post on why you think RAG— it's multiple predictions, but the one that jumped out most to me is why you think RAG will trump fine tuning. And here I wanted to just super quickly sketch an idea that I've been chewing on for what the drop-in knowledge worker of the future might look like and kind of just get your reaction to it. And then the last one is just future of the software industry. Like, what should junior developers do as well? So drop-in knowledge worker, we've covered the RAG stuff. Like, what's hard for the models today? They don't have context. And I always am so, like, I feel bad for them in some ways because when I'm searching through my Gmail or my Drive or my codebase, one huge advantage I have is I kind of know what's in there and I know when I've found it. And in contrast, the models today just get what they get and they sort of have to do the best with whatever is returned, right? And you can turn up hyperparameters and that helps, but they don't know in general, have I found the right thing? Should I keep searching? And so I have this sense that continued pretraining is maybe one way to describe it. On a company's proprietary data. Basically try to get to the point where the model knows the company from an inside perspective, as well as the models today kind of know the world at large. And then kind of continue with your post-training, your behavioural refinements, but try to get to a point where the model knows, yes, I actually found what I'm looking for. This is the ground truth that I needed to to go on this task or not, and therefore I'm going to kind of keep searching and maybe using different tools until I actually get there. What do you think? Does that seem like too far-fetched, or how would you just generally react to that vision of continued pre-training? So I know, yes, I found it, or maybe that is not necessary for some other reason that I don't see.

Yeah, so I can say there are a few challenges with that approach. So one challenge with the continued training is that Even though— so training is very sample inefficient in that you need a lot of data for the model to learn something. And typical company knowledge bases, they're too big to put in the context window, but they are not large in the sense of training datasets. They're actually typically pretty small. So if you think of a typical code repository, it's not a lot of data to train on. And so if you want the model to pick up on what's there, you're going to need to do multiple epochs, probably train on it multiple times, but then you quickly overfit, which you also don't want to do. You don't want it to just memorize what's in there. You want it to actually learn from it, sort of. So I'd say one challenge is the amount of data for this to be effective is typically too small. Another challenge is keeping it up to date. So RAG, you can work hard, but then make your RAG solution instantaneous, or you can reduce the delay as much as you want. Training models is— there's more friction there if you want to do it for email, for example, every user has their own email store. So if you're going to be training a model for every user and keeping it up to date, that's again, there's not that much data. And the logistics of doing that separately for every user is tricky. I think it used— but I would say it used to be like, do you do RAG or do you do this? These days, I would honestly try to solve this problem with an agent who tries several approaches until it thinks it finds the answer. Like, it doesn't have to be just, we'll do retrieval and did it get it or not? You can do a lot more now. And so we've actually built a lot of that into the product. So the more advanced versions are not just like a one-shot retriever. We do more to give you the best retrieval quality we can. I would, just in the sense of simplicity, models are so good now that I would reach for solutions like that. And try them definitely before doing fine-tuning.

Yeah.

Yeah. Interesting. So you think you can basically just, in short, get good enough performance without ever having a model that really knows in a confident, intuitive sense, like I do, that it actually has found the right thing?

Yes, I think so. And what we're seeing, kind of, I'm saying that kind of based on the evidence, at least how I interpret the evidence that we're seeing. Yeah.

Okay, cool. Okay, last one, and it's not an easy one necessarily. We've got people talking about superhuman coders within this calendar year into next year. I'm referring to obviously Dario there who said that repeatedly recently. I guess I wonder, do you buy that that soon? Or even if you extend the timeline a little bit? And if so, what do you think that means? Or what advice would you give to people that especially are early in their software career today? From what I see on the internet forums, it seems like people that are just coming out of school with a CS degree are sort of like, yikes, this is not what I thought I signed up for. And not everybody can just pivot into being a machine learning all-star. That's like a great option if it's open to you, but for the rest, who are like, I did this because I thought I was going to have like a nice stable career with like a, you know, solid income and like, you know, never have to worry about my employment status. Where do you think we're going and what advice would you give them for navigating the challenges that might be coming for them kind of soon?

Yeah. So first, I think it's good to separate the short-term from the longer-term. So I think in terms of what Dario said, the way I understood it is if you look at what actually like in the near-term, You go to a new line of code and you ask, what actually generated this line of code I'm looking at? Then his statement, very likely it's going to be a model or an agent that did that rather than a developer. Yeah, I buy into that. Like, maybe it's not— he said something like 3 to 6 months. That's probably too quick from the adoption that we're seeing, but not like 3 years, I don't think. I think like shorter than that. However, that doesn't mean that the model decided on its own what to do. It doesn't mean that it supervised its own output. It doesn't mean that it's fully autonomous. I expect for a long time there's still going to be a developer there steering the model. And I expect that because this is how I work. This is how I see people who are picking up agents work. You look at their code, the code was almost— once they pick it up, it was almost entirely generated by an agent. But if you took away the human, nothing good would happen. Like you wouldn't get anything useful out of it because the models are, they're nowhere near that good. They're not even good enough to say, here's the product requirements, go build this. Like we're definitely not there yet. So I think I buy into that statement, but that doesn't mean we don't need software developers like in the next year. Now in terms of advice, so I have 2 kids, they're 7 and 14, and we're having discussions with the older one on like what to go for. My advice is to go for a career that's more tied to the physical world. Could be mechanical engineering or robotics or something like that, where it feels like it will take longer to be disrupted because it's very hard for me to predict what software development is going to look like in, let's see, 3, 4, 5, 6 years, something like that. That's very hard with the rate things are changing. I don't know where it's going to land. I think we'll still need developers who understand the system, because if you're just vibe coding your enterprise software, you will run into trouble. I mean, I can already see it happening with the code that I'm writing. It will get better, but I don't think it's going to get better at that scale. But then still the question is, well, how many developers do we need and how much software do we need to write? And I don't know. So those are the discussions we're having with my older one. With my younger one, we have a bit more time to figure this out. Maybe at that point we have AGI and everybody can just do art. I don't know, but I'm glad we have a bit more time to figure it out with him.

Yeah.

Yeah. Well, the local artisanal economy, yeah, could be a beautiful future as long as everybody has their basic needs met. Are you guys like, how does this get operationalized for you in terms of your hiring? Like, are you hiring junior developers at all? You know, is there any on-ramp for somebody out of, you know, a CS program to get into a frontier company like yours?

Yes, there is. Yeah, for sure. We look for excellence. We hire junior and senior developers. I think this is still a time where not— there's going to be a learning curve in knowing how to extract the value out of these models. And so it's not that everyone immediately becomes— even if you're using agents, getting value out of them takes time, especially in an enterprise environment or in a codebase like we're not— we're We're small. I wouldn't call us an enterprise, but even in our codebase, which is, let's say, small to medium size, using agents to navigate that codebase requires some skill. So I think for a while there's going to be ramping up time where it seems that the people who, you know, as always, people with less experience are also like quick to jump on new technologies. So I think we're going to see a lot of that in the near future. But yeah, short answer is we do, we certainly still hire junior developers.

Great.

Anything else you wanna leave folks with before we break?

So Augment is out. It's really good at understanding your codebase. So I encourage you to download it, give it a try, and really feel the power of an excellent AI assistant that fits into how you work.

And I would definitely encourage people also to check out the blog for a bunch of deep dives We didn't even get into the inference optimization work and all the detailed analysis of batch sizes, which I did think was super interesting. And there's a great write-up of the next edit feature as well. So there's plenty more to be unpacked from the Augment team than we've had time for today. But nevertheless, this has been a great conversation. I really appreciate it and look forward to continuing to play with the product. For now.

Yeah.

Guy Gerary, co-founder and chief scientist at Augment. Thank you for being part of the Cognitive Revolution.

Thank you so much. This was a lot of fun. Thank you.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.