How to Evaluate Coding Agents

Welcome everybody. This session is about evaluating coding agents. Now coding agents are a little bit different than what we teach in this class. Coding agents tend to be more closer to foundation model benchmarks in the way you might want to think about evaluating them, less like product-specific evals. There is like a lot of interest in this class about coding agents. It comes up all the time. My friend, he has a company called Memento, and I'll let him introduce what Memento is. But yeah, it's an honor to have him in this class. And yeah, thank you for coming and showing us what you're working on. And and how you're evaluating it.

All right, thank you, Hamel. Thanks for that intro. My name is Kwaja. I started my career working on cameras onboard the Mars rovers and started really working with scientists. And then I kind of converted into a distributed systems person working on high-performance things and eventually made my way back into the inference era back 15 years after I was initially working on that space. And it's really brought back the memories of what it's like to be a scientist. And today I'll talk to you a little bit about what it takes to build an eval system for coding agents. And I just want to preface this with, we started going down this path a little over a month ago, and it has been nothing but a humbling experience on how difficult it is to build an eval system, let alone an eval system for— Coding agents. a coding agent. And the— that humility is really, really important because the moment you lose the humility is the moment where you get, you get overconfident and you stop diving deep into your entire eval setup. So I wish I could come here and tell you that I have all the answers on how to build an eval system for coding agents. But I can share a lot of things that we did wrong along the way and what we're learning. There's still a whole bunch of open problems that we're trying to address. And when I started working on evals for coding agents, my initial instinct was, well, we built a bunch of stuff and we got a bunch of really, really cool soundbites. Like GLM 5.3 Flash does the same task in the same amount of time with this provider for $0.20, whereas Astra is taking $9.28. Job done. Like eval system is complete and we're ready to go. And as tempting as that may be, you know, we could just end the talk there and say we're done, or we can kind of keep diving deeper. And this is kind of like the repeated theme of this talk is when we kept Diving deeper and deeper. We started thinking about, well, what does it take to build a rigorous eval system? And you're trying to create a set of tasks, and it all centers around the tasks that we are trying to use to assess the efficacy of a particular agent. And within that task harness, as we come up with this eval system, you've got to generate tasks. You gotta think about how am I gonna grade those tasks? You gotta do sandbox execution with a lot of repetition and all of that feeds into a score. And the score, the punchline there is like the score in our experience so far, like we haven't converged yet. Every time we see scores, we are ending up with more questions than answers. And I'll talk you through why, what some of that looks like. So let's say you started by thinking about, okay, I'm going to go eval this entire setup. And that entire setup has a, you know, there's a human, there's the prompt, there's the actual coding harness, there is the model, and then there's the various reasoning levels for a given model. Each one of those things is a pretty big variable, and that entropy kind of just multiplies along the way. And any one tiny tweak to any one of these things can make a big difference. But let's talk about how do you start by selecting what tasks you are going to evaluate the entire setup on. And you can start by SWE-Bench or one of the existing kind of benchmarks that are out there. And, you know, If you're an enterprise looking at a variety of coding setups, the SuiteBench may not be, or whatever benchmark you find, may not be representative. Like, you may not be a Django shop or you may not be a Go shop. It's not representative of your style, your frameworks, the types of problems that are happening in your repository. It's been public for years, so guess what? All the models are already training on it because the criteria is out there. And it has somebody else's definition. Like, you may have a very different style and a different definition of what done means inside of your organization versus what the SWE-Bench and existing other benchmarks are doing. So, okay, now we say, well, let's go look at our own repository. But how do you select the tasks from your own repo? The most obvious thing is you say, well, I'm going to go find I'm going to go through my PR history and I'm going to find a place where I fixed a bug. You find something that's super easy to measure. There's a unit test that's failing and I want to see which setup can fix that bug. I'll just evaluate, did it flip the test and how long did it take, how much it cost? You can do that. It's just rollback a version and Let's see if the model can come up with an answer. And by model here, I really mean the whole end-to-end setup. And it'll give you an initial set of tasks that you can do that, that you can test against. And then there's a design fork here where you really have to decide if you want the tasks that you're generating to be representative of what you're doing, Or do you want them to be measurable? If, like, the example I gave is incredibly measurable, right? Did my unit test pass or not? Like, it was failing before, it passes now. The definition of done is very, very crisp. You can also go a step further and say, well, I want to make sure that my other unit tests didn't fail. But that is extremely measurable, but it's not necessarily representative of a day-to-day coding exercise that developers might be going through. The other tasks, like building representative tasks, requires a lot more non-determinism and a lot more uncertainty. And those are actually harder and harder to measure. So these things are at tension with each other in terms of whether you make your tasks representative, whether you make them measurable, or you do both. And it's a spectrum. There isn't really a right answer, and you really need a full kind of combination that is specific enough to grade and real enough to matter. And as you're going through that loop of identifying the task, grading them, running them, diving deep into the traces, you're constantly thinking about, well, what do I need to do to enrich my eval setup? And our initial prompt— so the other theme here is that you constantly have to just go back and keep digging in. Right. Like, what happened? Why did it happen? And we— that big headline number, that was generated. It was a, you know, we just rolled back a particular fix on the Valkey codebase, open source repo, lots of tests, easy to find issues that are fixed and the unit test that went with it. And as we went and looked through the prompts that was generated to kind of go solve this problem, the prompt looked like this. It's like, Big headings, here's your task, here's what you need to implement, here's the source, the file that you need to go change, and here's the test that you need to go fix. There's a whole lot of problems in how representative this is. First of all, it names— this might be a little hard to read, but the implementation file is named for you. It's like, go fix this. It's like being in college, like if you went to a code school or like a CS101 where the teacher has created method stubs for you and you just have to go fill them in. It's much, much easier, right? Because it's telling you, here's the file, here's the test you need to fix, here's the grading command. There's minimal ambiguity. And the other reason why it's not representative, it's like very few people in the real world— I'm pretty OCD about this, but most people end up running very, very long sessions. So the session history is diluted. Over time. And the context is diluted. You don't show up with a cold session with a well-defined prompt like this that's really, really well laid out for the agent. There's all kinds of crap beforehand that may or may not be relevant, and they may actually distract from the solution. And no human, or got to be a pretty maniacal human that writes prompts like this. We just talk to it. We just talk to the coding agent. So as we started getting into this, we then started thinking about, well, what does hard mean? We wanted easy tasks, we wanted hard tasks. What does hard really mean? And there's somebody on my team for whom— and for him, running a 5-minute mile is very, very easy. And for me, that's very, very hard. Right. But there are problems that I can solve in ways that he can't solve, and there's problems he can solve that I can't solve. Each model, each human is optimized for a specific set of things. And what looks hard to you as a human may not be hard for the model. And it's really important to understand that particular delta there. But what really— some things that make life harder for a model is always longer context. All you need is attention and, you know, like you— your attention gets diluted over time as the longer the context gets. And the longer the context is, the less focused it's going to be. But if you have a longer context that often kind of conflicts with each other, with itself, because that happens, like if you, you know, and you can go out of your way to put traps in the context itself. One example would be if you have like a team meeting notes summary where you went on a digression and had like a completely wrong way of doing it, that could trip up a model. Especially if it's on high reasoning and if you need to make a lot of tool calls. Those are the kinds of things that make life a little bit harder for a model. It's not about solving— you can give it a very hard algorithmic problem. Most of these models are actually quite good at LeetCode-style problems that may appear hard for humans. So now let's get back into the grading side. The super measurable tasks are easy because you can run tests on the whole repo. You can make sure that the test flips. But you can also— this is the hairy one. I know it's one of Hamel's favorites, is you can just lean on an LLM judge. And the problem here is that the more measurable a task is, the easier it is to not rely on an LLM. But as the tasks— as you start adding entropy, you might end up writing prompts 5 different ways because humans aren't going to write the prompt the exact same way. Like, I won't write the same prompt The same way 5 times if you ask me different times of the day. As you start adding more entropy and as you start adding more variability into the task, an LLM judge may be required to augment the specific measurements that you may have. Now, this is where getting into how do you rely on this judge? How do you do inter-rater agreements to make sure that the judge is actually agreeing with you? There's a whole lot of can of worms here. That open up as you start to rely on the LLM judges. And the other part of it is, at least we found that we're trying really hard to not let the agent detect that it is being judged, that it's undergoing a test as well. So if you— the moment you give it a very crisp grading criteria, you've kind of leaked that, you know, it's under a test. But That said, when you're doing a bug fix or you're making a code change or you're making a performance improvement, you can detect the bugs that you thought about, but you can't really prove the absence of the bugs that you didn't think about. And that, again, some of this stuff just doesn't show up. Even when humans are coding, when you build mission-critical systems and deploy them into production, no amount of testing catches every problem. And there are controls that you have in place that get fed in, and then you can do rollbacks and so forth. So you can never prove the absence of the bugs or the completion of the task that's bug-free, but you can detect when there are very obvious bugs that you have thought about. So we started going down this path of, well, for the more The less specific, less measurable tasks, what we call the more representative tasks, we needed ways to judge the completion and assess the setup's efficacy on them. That requires us to build an eval for our eval. It's quite meta to make sure that our judgment criteria are reproducible. And then When we run these tests, we realized coding agents specifically, they stress the models in ways where the models are forced to get very, very creative. And we learned the hard way that the level of sandboxes that we needed was always more than what we thought it was. So when we initially started doing this, we rolled back Git, we gave it, you know, gave the setup a task to go complete. We observed in some cases it just rolled the history forward and found the answer. And we're like, wait, you can't do that to me again. I'll do a Git snapshot and give you only the point in time where you can't roll forward anymore. So the second time in the early days where this happened, it actually found a completed answer from from another agent on the local file system. Harnesses have built-in things to remember, and especially some of these coding harnesses. So they have ways to recollect things. And it was kind of spooky when it went and found another answer. And you wouldn't have known initially. You can just say, oh, look, it did it for cheaper and it did it so much faster. But nope. What happened was it went and started looking at other sources that had been done. So even the grading criteria, even when you tell it, like, here's how I'm going to assess whether you're done or not, the definition of done gives it hints as to what it's going to take to succeed at this task. And it is very different than how human coders are prompting their agents. They're not that specific on success criteria. They're usually like, go fix this bug, or go add this, go move this button or change the layout here. So agents should not know that it's being tested. We try to make the environment as representative as possible. We try to minimize any score files in the workspace. So we run it in a sandbox, it's in a Docker. We contain that Docker with minimal network access. We're kind of instrumenting every network call and tool call that's being made. Gotta make sure that session memories aren't feeding in or memory breadcrumbs aren't there. And prompts have to read like normal human prompts. We kept learning that the agents were very creative and they kept breaking our isolation every time. And while we kept getting better at it, we also realized that you cannot in fact sandbox everything. But as we went down this notion of sandboxing and finding ways to isolate, what we learned is that there are things that you just cannot sandbox. So like if you have deterministic tools like your compilers, your test runners and so forth, those are easy to sandbox and isolate. But coding agents often require you to make network-facing tool calls like Sometimes they'll do a web search. Sometimes if they're connected onto your wiki, you know, on your Confluence, like they will make calls that are really, really hard to contain. And Google might give better answers this week than next, or it, you know, it may give better answers next week. And that entropy makes things really, really difficult. And again, it kind of goes back to the measurable versus representative task. Like coding agents inherently end up doing a fair amount of web tool calls. And constraining that environment becomes really, really challenging. So there's 2 ways that you can kind of get around this problem. The first one is just repetition and aggregation. You just keep going and you keep running the tests over and over and over again, and you kind of understand what's happening along the way. And the other one is you can, like, so repetition is one thing, just get as much data as you can and then going deeper into traces, like continuously looking into your traces is another really crucial thing. Like what happened, why did it happen? Having that intimate knowledge of how things are shaping, like why models are failing. Oftentimes we kept getting Really, really bad signals. Like we would think that the test at like a particular model had failed, but we realized that we were just being throttled by the inference provider, right? Like if you don't like dive in to see that's what happened, or you got an error on the gateway, or a call just hung, unless you dive into why the tests are failing, and unless you have enough repetition to kind of facilitate that, you're not gonna get an answer. So these are like, they're like basic things like the whole like pass @k and pass to the caret k. Like it's essentially knowing how reliable and how repeatable the agents are becomes really, really crucial. And with coding agents, like it really comes down to what is your environment like? How well can you detect when the task isn't actually done? And because if you can detect when the task is actually done really, really well, then even if a really, really cheap agent can solve it 1 out of 10 times, you can just run it 10 times and pick the answer, right? But if there's a lot more ambiguity, then you need more repeatability. You need to be able to consistently pass the test over and over and over again. And you need to understand why things happen. And we— We constantly kept learning that a single run is not a measurement. There were scenarios where we did this experiment where we were like, okay, let's run this particular task across 2 agents and pick the best for one agent and the worst for another agent. And there it looks like the good agent did 6.6 times better, whereas if you aggregate it, there was only like a 16% delta. So So look, marketing-wise, it's really easy to come up with numbers that look flashy without really being scientific about how you're kind of getting into the numbers. And then as you look at the aggregates, instead of just saying that here's the variability that you saw, just constantly diving into the traces to make sure you understand why and can attribute it becomes important. The other part, when we measure our agents and our setups, we're looking at, as I mentioned, the human, the prompt, the harness, the model, and the reasoning level. And we're looking at dimensions along the lines of speed, correctness, cost, and policy adherence. Those are the 4 dimensions that we really tested on. And we fell into this trap where we were looking at how long, like we were just measuring the latency. How long did it take an agent to complete this task? And a lot of it was diluted because the grading criteria was taking, like the running all the tests was taking 20 minutes and the task itself was taking less than a minute. So it looks like all the agents are performing at about the same speed because Well, 19 of the 20 minutes are going towards the grading. And until you dive into that, you really don't know, and your results can just remain diluted inside of that. The other lesson that we learned along the way was your harness needs to have observability built in. I've been a pretty big fan of the Strands agent harness that Amazon just released, but one of my pet peeves on it so far, and we might just go add this, is like, It doesn't have the level of observability that we need to grade how it's doing. It doesn't show you the cost or how many turns. Like, we have to instrument a lot of it outside of the harness. And my advice here would be just like, make sure that your harness is emitting the right metrics, emitting the right traces to make your debuggability and your evals a lot better. And these investments go a long way. All right. Just keep in mind, again, while you keep diving deeper, every little component makes a huge difference. And the entropy is just very, very multiplicative here. For us, the model matters, the harness matters, the provider or the gateway matters, the policy and the reasoning levels matter. And any one factor changing makes a pretty big difference in your end-to-end results along the way.

Well, yeah, thank you so much. I think everyone really enjoyed this to like hear your experiences with, uh, creating evals, especially for something as difficult as a coding agent. I hope people give MoEvals a try. Thanks so much for coming.

Awesome. Thank you guys.

Real quick, if you're building an AI product and you're stuck on how to improve, we put together this free resource on evals to help you. It draws from more than 1,000 hours of working with companies on building AI products, and it's customized to what stage you are at. So this is how you can use the guide. Go to bit.ly/evals-guide. When you open the guide, it'll look like this. And what you do is you select the stage that you're at. Maybe you're new to evals, you don't even know what evals are. Maybe you do have evals, but you don't really know what to test. Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval, or it's too time-consuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. We've worked on it for over a year and are giving it away for free. Go to bit.ly/evals-guide and we'll send it to you.