How To Test AI Agents With Simulations
Hamel Husain and Raindrop CTO Ben explain why traditional input-output tests fail for AI agents and how to evaluate them instead by replaying production traces in simulated environments that recreate the original tools, databases, and APIs. They cover diff-based metrics like answer changes and latency, sample-efficient trace selection, and when simulations are worth building.
Chapters
We've been running this evals course for a while now. And every so often we bring people in who are working on evals. And there's a lot of different vendors in this space. And I have used every single vendor and I try not to have like pick favorites, but I really do like Braindrop. You know, I like their philosophy, like how they go about things. They tend to do things that are very aligned with the way that we teach in this class. And so I think it's really helpful for you to see different aspects and things of evals. We thought, okay, let's invite Ben to give like more color around the way that he thinks about evals, what he's working on, et cetera. I think it would be really beneficial for y'all. And so yeah, I'll just go ahead and kick it off to Ben.
So talk today is called Getting Started with Simulations. I'm the CTO and co-founder of Raindrop. This is both a product that we recently launched, also something that just in general, everyone in the space is starting to do. Labs are really starting to do, writing papers about. It's not not evals. We'll get into it in just a little bit. Our whole company is about finding critical issues in production agents. So we find those issues in real time, like minutes after they start, verify those fixes actually work without side effects. And then now we simulate those changes before they land in production. We also have this really cool open source tool, which honestly, especially if you're ever hacking on your own thing, it's a really nice starting point. I use it even in a lot of my own side projects. I'll just be like, hey, take this UI, Either use it, change it, whatever you want. It's kind of cool. Of course, the tool that our Fortune 100, Fortune 500 customers use is what we call Raindrop. We also make this really cool guide, very, very Hamel-inspired, called howtoeval.com. It's probably due for an update even though we just published it 2 or 3 months ago or something like that. But the space moves fast. Everything's changing, which is why these courses matter. Everything has changed to agents. This isn't just a word changing. It's actually kind of a big It's a big deal and I'll kind of explain why. In this kind of chatbot era, the way that we would think about evals— and again, the word eval is very overloaded, so I don't mean to say that evals are dead or something like that, but just what we used to consider— the way to eval has changed a lot is the right way to put it. What we used to consider an eval, maybe a little over a year ago, was you have a prompt, it goes to the model, you have some sort of response that comes out, And then you have some sort of predefined way, probably for that given input, to know that that output is correct. A very contrived example, but an example that you'll still see on vendors' websites even now is, what is the capital of the United States? And then it's like, oh, well, it's Washington, D.C. So it's input-output validator. The thing though is that things have changed a lot since then. One is that agents are a lot more sprawled out.
Right.
So it's no longer that you have this nice clean Oh, well, you just have to, you know, you change your prompt and then you rerun and you see what happened. It's like, well, now it's like you have skills, you have the tool definitions themselves. Like just adding or removing a tool can completely change the agent's behavior. Even just changing the input type to a tool can change, or even the field name to a tool can change how the agent behaves, right? So it's like, clearly it's a lot bigger than a prompt now. You know, users connecting MCPs, you have, of course you can change the model, et cetera. But then now you also have memory. Yeah. You have subagents and then the prompts for those subagents. A lot of these also increasingly are outside of the repo. So you no longer have all the context even in just the repo, which is really interesting. The other thing which is related to that last point is agents are very intertwined with the world now. They have all their little fingers and arms out touching the real world. Whether that's your own sort of internal production databases, whether that's other services. And again, no longer is all of the state and all the context, like the emails that the agent goes and reads, are now part of the context. That's part of the context that's influencing the agent's behavior and what it's going to do next. That makes them a lot harder to eval. And also, they've become more and more capable, more and more complex. And what that means is that sort of like single input to output pairing.
Yeah.
Really starts falling apart. Because most of what your agent is going to do is, and most of where it's going to go wrong, truthfully, in practice, are going to be things that you haven't really thought about yet. And that's what makes it really hard. And so I think that there was this, like, for some time in the evals world, there was this, like, more is more strategy with evals, which is like, the more evals you have, like you have 4,000, 5,000, 6,000 evals, and then that is gonna like, That's your moat as a company. That's going to be the thing. It's like, well, there can be valid cases for that. If you're planning on training your own frontier model or your own model, there are cases, and we can go into more details in Q&A, where that really does matter. But I think largely it's actually more about your strategy around evals than the number of input and output pairs you can collect. Because again, capabilities are increasing. So essentially, Your agent can do more things faster than you can make evals. And the other problem is that you start running into this problem where it's like, okay, so now I have 4,000 evals. Let's say if I add a tool, am I— let's say I add some sort of tool, am I actually going to run 4,000 evals in that PR? Probably not. So then it's like, well, how many are you going to run? How do you find the right ones? I think this is why as a company, we actually stayed away from evals for a really long time. We were kind of like, we focus on the real world because That felt like the best agent thing to do. It let us catch issues you were never expecting. It let us have a lot of signal around those issues. For example, one thing we saw with evals was that the harness would change and the evals would break. The evals were expecting a specific tool to be called, but then you switch to a different harness and now everything breaks. The reality today is that even some of the best companies in the world AI-wise barely have functioning evals. I've seen this time and time again. I won't call anyone out, but Even some of the poster child of evals company-wise, you talk to them and they're like, yeah, well, all our evals broke when we switched to Claude Code as the harness underneath or something like that. So it's actually really, really interesting. And so we really focused on production monitoring. And we did that essentially, we think of it and we call it unsupervised real-time issue detection. So it's like there's 3 ingredients that go into that production monitoring. The first is your codebase, your prompts. So it's like we can sort of understand based on how you've prompted your agent, based on the codebase, what should be happening. So that kind of gives you one data point. The next is the baseline of behavior. So something like you make a change and it drifts, that's an interesting signal. And the last thing is your feedback on the production data. So looking at traces, I know Hamel emphasizes this a lot, Look at your traces. We'll send issues, we'll send summaries, we'll send alerts. And based on how you interact with that, we can learn what you care about and what you don't care about. So again, this is sort of how we've figured out how to do that eval, but in production, right? How to really, really high signal when almost every single issue we send a team, they react to, they open, they do something with now, which is something we're incredibly proud of. And I think when you use other tools and you're just getting inundated by Random digests and summaries. It's like, well, that's not useful. But this is just sort of like, again, we do these deviations on— we figured out as a company how to do these deviations on not just error rates and things, but things that are very semantic, like understanding unusual patterns of the way that the agent's responding, the trajectories it's taking, etc. This is what an issue looks like in RangeOps as well. Not to hammer this home too much, but one thing that also I think teams really get wrong— somewhat of a tangent here— With issues and this sort of thing is just show the traces. Have tools that show you your traces. Do not drown in AI slop summaries of what your agent's actually doing. For example, here we show these little annotations, these comments on the trace that kind of helps you jump to the right point, which is important, especially now that traces can be tens of minutes, if not hours, if not days. But just don't get lost in the sauce of AI summaries of what happened or what went wrong. have ways that accelerate finding the right traces and help you understand the trace that you're looking at. I just talked a lot about production, but the thing is that that is nice, but it doesn't prevent issues from happening. It's good to catch an issue as soon as you can. It's good to know that your change actually did something well in the real world. And certainly the real world is a really powerful arbiter of truth. Again, it's not like, oh, your harness broke something and then your old eval you defined stopped working. It's like, People are complaining. People are mad. Or, no, it's really working. Look, you can see people have recognized the difference somehow. But again, it doesn't prevent issues from happening. Especially as we have Fortune 100 customers, as we have even defense. There was a story recently about the US misidentified a Chinese vessel as carrying nuclear weapons of some sort and intercepted it based on bad information from an AI. We don't know which one. Clearly, this is getting higher and higher stakes as we go into healthcare, defense, finance, etc. As a company, we're like, how do we actually prevent these things from happening in a way that is not ridiculously fragile as well? I think whenever you're faced with this sort of question, it's really useful to take a little bit of a step back and say, what would OpenAI do? OpenAI, or insert Foundation Lab here, but they're a big company. They have a lot of users. They have a pretty good product that's pretty stable. Codex, for example, I think is one of the best AI products that exists in the world. So it's worth asking yourself, well, what do they do? This is one paper recently on Codex. They've had a few around simulations. Essentially the answer— I kind of gave it away actually— is simulations. Every time they make a change to a model, they have a whole rig that lets them simulate the past traffic on Codex and say, how would this new model change the previous Codex trajectories? The really powerful thing about this is that you're now answering the question of what would my change change? I think that's a lot of times the rut that we see teams get stuck in. A lot of times the fixes and getting to the point where you have a fix to some sort of problem with your agent or add some sort of capability, I think getting to that point is actually getting faster than ever. But then there's this whole consequence of making that change. And that's the thing that really starts slowing teams down. So really simply, again, simulations are you replay real traffic, real historical traffic against your new change and you see what's different. So that change can be anything, right? That could be a new model. It could be a new tool. It could be removing a tool. It could be changing the prompt. It could be like you switched from AI SDK to Pi. It could really be as big as you want here. And so you're sort of like, again, you have the old traces that you've collected and you find a way to just rerun your agent on those exact traces. So again, what will my change change? Does it actually fix it? Does it cause other issues? Is kind of what you're asking. Let's just take one very specific example. Let's say you delete a tool. So what you can do is find traces where before you made that change, that tool was called. Real user events where that tool was used to answer their question. And then replay those and then say, does the answer change? Right? And then if the answer changes, you know, flag those and those are the ones you want to review. So this again becomes a way that is like, one, extremely targeted to whatever change you're making to your PR or like whatever change you're making in your PR. And like, again, it really, really helps answer this question, like, what does my change change?
So when you replay traces from production, let's say, you know, your agent is like, has some multi-turn stuff going on. Users ask you follow-up questions. How do you deal with that?
This is a very, very good question. So this is, I think, one of the things that is actually very powerful as opposed to the way that most people think about evals. A lot of times we see teams do these single-turn evals. And I think the single-turn versus multi-turn is actually a very somewhat convoluted term right now. So there's multi-turn as in you're 20 messages into a conversation and then you ask the next question. And then there's multi-turn as in You see the output and then do another question, see the output, do another. I think the former of those doesn't really work right now, just to be 100% honest. I think that there's people trying to work on user simulations, as in more user prediction. And I think that's a really hard problem because you take something as simple as a user looks at the UI and they're like, oh, I don't like this design. It starts becoming a pretty hard thing to simulate.
Yeah, it feels like AGI. You have to simulate the entire existence.
It's almost more.
It's like a whole human.
Yeah, exactly. We need to clone Hamel in order for that to work. What we're talking about here is a little bit different. We talked about earlier that the agent's really sprawled out. The agent's really intertwined. It's intertwined with all these production databases, all these services. It can talk to Slack. It can talk to GitHub. It can talk to all these internal databases. How do we say, you add a new tool or you move to a new change management? We simulate that entire world around the agent that as best as we can that existed when that agent ran in the past. All those services, there's an illustration in a second that'll explain how it works. But that lets you test those historical traces in a harness-agnostic way. And in a way that's not like, I think the naive way of trying to do this is like, well, you have the old trace, why don't you try caching it, almost using those as cached results? So when you rerun it, the agent calls a tool and you just give it the same results. Well, Actually, if you try to do that in practice, the agent will instantly call your bluff. It literally, like, you can try this, it'll just be like, I'm in a simulation. This isn't right. And simulation awareness is one of the things we measure here. Does that answer your question? We'll talk a little bit more in detail.
So what you're describing with simulating the tool calls and stuff like that, is a good way to think about that some kind of agentic mocks? Simulate a service? Is there an LLM behind trying to simulate the data that's coming back? That's really interesting.
Yes, exactly. We think about it in a few steps. The OpenAI paper is also a good read here, but there's really 2 or 3 steps. What we think about is that it's a little bit like Swiss cheese. In the original trace, there are tool calls and those tools had results. Every tool call is almost like a little window from the agent into the state of the world at that time. One thing that's really important to remember here is that the world Let's say I'm replaying a trace from a week ago. The world has changed since then, right? So I actually can't just like, even if I could just connect my agent to the current production and use all those databases, don't do that, first of all. But second of all, even if I could do that, you wouldn't want to because the state has changed. Take something as simple as a customer support bot for a delivery app. It's like if the user asks, well, how far away is my order? And it says, well, 15 minutes or something. If I try resimulating that and it was hooked up to the current production database, it would give a different answer or no answer because nothing is pending. And so you really do have to understand the world. We take a 2-pronged approach. One is that we try to upfront generate both using the previous tool called Results and any sort of transformation we have to do for the current harness. And you sort of want to pad those holes so that if it's slightly off or round, it'll get the right result. And then also at the same time, we're evaluating the quality of those responses and the agent's awareness that it's being simulated. And you can also do that once it's complete. The OpenAI paper talks about how they will essentially take the original trace and the new trace and then ask the agent which one is most likely to have been simulated. And if you do that at a population level, you can sort of detect it that way. And so essentially, it's kind of a nice hybrid between having enough data upfront where you're not slowing everything down, but also you have an LM that's always there to spit out the correct data when possible.
Can you tell me a little bit more about the simulation awareness? So are you saying if there's this situation where your agent will just refuse, it'll just say, hey, I think we're in a simulation. Haha, nice try.
So refusing is one, but there's more where the agent can be like— and I think one of the insidious ones is like— and to be clear, this applies to all sorts of evals, right? I think eval awareness in general is a very big topic right now. Because if you're like, imagine you're doing safety evals, and if the agent realizes it's being eval'd, it's like, oh, well, I'm being eval'd. I should probably not do this, or I probably should do this, or I should probably behave in this way. And it'll learn to do that. There's a similar thing with simulations, which is like, you want to make sure that the agent does not understand, believes that it is in the real world and that there's nothing really differentiating in that way. And that shows you how good of a job you did simulating. I think in general, this is something that has changed a lot in the last year or two, is that models now are trained to understand what evals are, how to do them, how to detect them, how to write them, all these sorts of things. Which also means they're self-aware and better at finding out when they're being evalled, when they're being tested, and finding creative ways. The extreme example of this is the Hugging Face incident. I don't think a lot of people realize that what was going on at that time was that they were running cybersecurity evals. They were just evaling their agent and it was trying to find the answer key to the evals that it was being run. That's kind of the most extreme example of this.
Do you see this happening in production at these companies you work with? Sometimes you find that there's eval awareness?
Yeah, definitely. Again, I think it's not usually like the Hugging Face incident, at least not usually. But It's more benign. It's like, oh, I see, like this thing wasn't working because it's a simulation. I should probably answer like this. And now you've just completely lost the whole point of the eval. Or like, oh, the agent, or like the user is probably like, based on this user question, the user is probably testing to see how polite of a response they would get in this case. Like, I should be courteous of my tone. And answer like this. And so it's like, well, you've just lost the whole thing. And to be honest, I think that this is actually much more of a problem in the absence of simulations in many ways. Like, team, you think about all of these tools your agent has access to, to search Git, to search issues, to search previous conversations, all this sort of stuff. If all that's empty when the agent tries to run, it'll instantly know that it's in a simulation. It's like, oh, I'm in a fresh environment. I'm in an empty sandbox. I know that, like, This is just all fake and the evals themselves end up being a lot less useful.
And so when you go to simulate this environment, like, okay, you're not simulating the human, but you're simulating like sort of the infrastructure. Are you recreating the infrastructure? Are you recreating like to what, or is it like kind of like an LLM computer that's like getting a request saying, okay, I'm gonna mock this response based on this and my understanding of the world. Or like, do you actually create all this code and all this like infrastructure? To what extent, or is it in the middle somewhere? Or like how does it— It's in the middle.
It's in the middle. We create as much infrastructure as we can. And the reason for that is that there's a lot of things that are somewhat deterministic. Imagine a common one is teams will have some sort of SQL dialect or something like that they have internally for fetching data or something like that. And it's like there's a parser there. So the agent's generating something and then it's getting parsed. As much as we can use the actual code to validate what the agent just did, that's what we want to do. And so we try to push that boundary as much as possible all the way out to network calls, essentially. Interesting.
That's really fascinating.
That's the really magical thing about simulations. Again, I think that there are ways to get started with this just on your own, depending on how complicated your infra is. But it really goes back to the looking at your traces thing. You want to see your actual real agent traces, right? Not some random AI slot thing of something. You want to see that your actual agent ran. The code for your agent ran and this is what happened. But ideally, again, you wanna do that before production is also good, right? Testing new changes.
Yeah.
So we talked a little bit about, come back, like you don't need thousands of simulations. OpenAI will talk about doing thousands of simulations. You really don't need thousands of simulations every single change. I think how many simulations you need is really a factor of like how big of a change you're making. If you're changing, like say you remove one tool, it's like, well, like. Maybe a little bit more than 5, but probably on the order of 5 to 10 traces that called those tools. You're sort of just sanity checking that the agent still does the same thing. If you're rewriting the entire harness or something like that, you might want to do 50. You might want to do more than that. It really, again, it's both how big the change is and how well you pick the scenarios. You can imagine that 1,000 wouldn't be enough if I removed a single tool if I never resimulated a trace that actually used that tool. And so again, it's really about how well you select those scenarios. And how big of a change it is. And some of the things you're looking for when we do simulations, and these are sort of just like, again, we'll talk a little bit more about evaling simulations in a second, but like things we're looking for is like, is the final response different from what it was before? Are there new tool errors? Are there new tool usage? Essentially, did the trajectory of the response change? Is the step count different? Like, is it now taking many, many more steps to do the same thing? Cost and like latency are the other big ones. Like essentially, is there a huge cost increase now? Or did the cost actually decrease as a result of this change? These are some of like the diffs that we're doing based on old traces. So one thing we think about are like, there's essentially diffs and then there's evals. So diffs are kind of what we were just talking about, which is like kind of like what changed. And you can also use an LLM to decide if the change is good or bad. But then there's evals, which are more like scores that you wanna like push up and target over time. I say score, not all, not just in like a, you know, 1 to 3 sort of thing, but like pass/fail. And the number that pass and fail. Back to evals. We sort of think about it like simulations are this base infra in some way. On top of that, you have anomaly detection. That's kind of like, did something weird happen? And evals are what we define, and I think increasingly are being defined as, pure functions that run over traces. As opposed to these input-output pair evals, which still exist, they're more like pure functions. And when I say pure function, what I mean is it takes an input, And it returns an output. So like that input output could be like a pass or fail. Also the input can be both the new trace and the trace before the change. So just sort of defining this really quickly. Like we talked before how there's— 2 years ago when we talked about evals, I think people mostly talked about like input output pairs. So people talked about like, oh, generate this React component and then it should generate code that's like this. Right. That doesn't scale well. When we talk about evals now, most of the time what we're talking about is, again, you, you might hear the word judge. You might hear the word score. You might hear like. Essentially, it's something that takes an input, which is a trace, and then decides to square back. For example, let's say verbosity. That's one that we see teams use. You don't want the agent to have these super long rambling responses. So your team might create a new eval, which is verbosity. And maybe either it's too verbose or not, or say a score of 1 to 3 or 1 to 5 of how verbose it is. I generally prefer the binary checks. But we can talk about that at a different time.
And is the right way to understand the one on the right is underneath, inside this function f , you're computing all these metrics like the diff, the length, all that stuff you showed on the previous slide, and that's informing the pass and fail?
Yeah. So for example, you could create an eval that's within 20% of the number of steps from before, or it's within 20% of the cost, or it's like the output length is within 20%, you know, or something like that. And so that's like, you can start to, or like you can create all kinds of cool things like this. Like you can say, is the answer the same? Is the answer more concise? And have an LLM decide like, is it more concise? Right. And so the key thing as opposed to the input output pairs is that this eval, once you define it, should scale to any arbitrary trace, right? So now it's like, this is something you can just run over traces versus like it being something that you're kind of accumulating these like input output pairs. The way to really think about it and how we think about it is you are aligning your organization, the humans in your organization, to a definition of what good or bad is.
Just to make sure other people are following. So Y in this situation is a trace that you've decided is good, correct? From the past, you said, hey, and you want to make sure X is still good. Is that how to understand it?
It can be, but it doesn't have to be. This is one of the really interesting parts. Certainly this could be a dataset of good traces. You could have a whole dataset. And you could run and say, essentially, well, if I replay this trace, I want the same answer. There's also often value if you think about the output length one. It doesn't really need to be something that you know is good. You want to say, oh, I don't want it to get more than— I want the output length to be within 20% of what it was before. I kind of know the state of my agent today. For example, let's say I'm trying out a new model. I'm like, oh cool, GPT-6 just dropped. Let's plug it in. And as long as it's within 20% of the length, of what it was before, then I'm good. I'm good to ship it. Or create an eval that's like, is it more verbose? Yes or no? And it's like, as long as mostly my pass rate is 80% of it not being more verbose, then I'm fine. Or if it's like maybe it's above 50%, you can decide. And so it can be good traces, but it doesn't necessarily have to be.
Makes sense. Okay. It's like one of the things you may want to check essentially is consistency. And so this is a— yeah, I like it.
I like this is a nice way to Is the answer the same is a good one, right? And then it also helps you narrow down the traces you should look at. Let's say that you create an eval that is like, is the answer the same? Run it over 1,000 traces and look at the ones where it's, they're not the same, right? Those are the traces you should be looking at. And it's either better or it's not. But again, creating something to score like it is a better answer. Well, that starts getting a little bit trickier. You could still do it. But like really easy one is like, is the answer the same? You could do that with Jev really cheaply. Is the final answer the same answer? Is it easier eval to create? Yes. This is again an easy one. To your point around the input data, it could actually be bad traces. That is one dataset we'll see a lot. Imagine a very easy loop we'll see in Raindrop is you get an issue notification. Oh, a bunch of users are complaining about this thing. They're not able to download a file or something like that. You push a fix. And you're essentially comparing the bad traces and saying like, does the new output exhibit the same issue? Right? The judge. If any of them return yes, then like now you know that you like your fix didn't work. So, so sometimes these are actually like bad input traces.
I like it. Yeah, it's pretty smart. It's like, cause this consistency thing is super important.
Yes.
Because this is a question that people want to know when they switch models or make a change and stuff like that. And I like the fact that there's a lot of metrics, like you don't necessarily need to call an LLM, like you highlighted The fact that you can compute all these metrics, which are super useful. And you might want to do that first before reaching for an LLM because those are super high-value kind of things that are not expensive.
Yes, exactly what you just said. I think there's a gradient of really deterministic things to you start introducing really deterministic classifiers to your— if you think on the completely opposite side of the spectrum, is the writing good or something? Well, man, it's a really hard and very tricky thing to define, which is why models are still really bad at writing. And so it's like, you can do it. It's possible to get an LM judge to break out what your organization means by bad, which is some mix of using specific phrases, using em dashes. But it's really hard. And so definitely you want to start with things as deterministic as you can and then start moving up the gradient of consistency, like you said. And, and then go from there.
Yeah. I'm gonna go ahead and front run this question cuz I know it's gonna come up. You mentioned classifier. Have you tried Gem in this? Yeah.
So we actually, we published this blog post like 2 months ago where we were talking about like classifiers, right? And called like our pipeline. And like, I think we realized like years ago, like, oh, you could actually just train really good classifiers and make it, you know, easy, et cetera, et cetera. That's not our business, but like we just needed to do it to make our product. And we've been evaling Gem a lot recently. It's very good. Our, our pipeline is still a, like I don't know, it's still like 50 to 100x cheaper or something like that. And the accuracy is the same. So it's like, but there's a lot of things that we're like now.
So you, so you did evals on your pipeline?
We did evals. We have so many evals. I couldn't even start. Like we have so many evals now. So many online, like everything. Uh, some, that would be a really fun one is actually like how, to be clear, like this is mostly like what I'm talking about here is not different from how we think about it, right? Which is sort of the magic of the company that we're building is like we try to just like be really upfront with ourselves of how we would do evals. A lot of it is, for example, historical data. Does it agree with what we've labeled before? If no, well, let's look at where it disagrees and have even another classifier decide if those disagreements are right or wrong. So we can then escalate that to Opus. You can start building those sort of chains where it's like, I only need to have Opus decide on the ones where it's it disagreed or something like that.
Another thing I like about what you're showing about this consistency thing is one of the things we talk about in the class is like how to sample data.
Yes.
And you want to be very sample efficient after you learn a little bit about evals. And so I really like the fact that, okay, you have this in your pipeline and you're trying to show these examples. These are like one of the kinds of things you're trying to surface people, say like, hey, this is probably something you want to look at.
Totally. Yes. I think about sample efficiency both in like obviously LLMs and costs and all that sort of stuff, but also like humans looking at traces. Like we talk a lot about like, you Look at traces, look at traces. Well, it's like, well, if you have like 100,000 events a day, like you're not gonna look at all of them. And so what are the ones you should look at? It's not even just the ones where things went wrong, because if you're looking at the same ones, you'll often have a few error modes like that dominate that you're not that interested in. Like if I think about a lot of consumer apps, that is like people complaining that the product isn't free. It's like, well, like, you know, that might not be the number one thing you wanna be looking at every single day. Yeah, very, very important from both perspectives. Again, just some more examples of easy things here. Is the tool call count after a model less or more? Super easy eval to define, really easy to run. We talked about verbosity a little bit, rating it between 1 and 3. This is one again where human alignment does really start to matter. Yeah, this is the loop. You log, you detect issues, you simulate fixing those issues, and you ship. And you ship knowing that at least having a little bit more signal that you actually fixed the thing before you ship, which is great. And then you can just do this loop over and over again. So just common footguns with simulations in general, this is outside the scope of our product. Our product exists to solve a lot of these things, but replaying against live systems, we talked a little bit about where that starts to hit problems and where the agent will start. If you try to do this, the agent will often start to realize it's being simulated. The easy one is there will be data from a week after when it actually was run in your production systems, the agent will see the fact that there's later data that disagrees with the time that was passed in and it'll be like, oh, this is no longer the right date. Just trying to serve cached tool responses doesn't really work. Only replaying failures is good, but you can overfit really easily. So you also want to be including a random sample when you replay. It's important to take into account these things around non-determinism. You want to replay it a few times, usually. Not looking at the traces is the big thing. Not looking at the traces is really, really critical. When you simulate, you want to actually look at at least a few of those traces. That'll actually tell you more than your eval suite will actually tell you.
I have a question about this. We talked about simulation. The nagging question I have in my mind is, okay, you have this agent. It might have complicated infrastructure. It might have tools. There might be databases. you know, all kinds of, all kinds of stuff. It may be very complex. And so how do you, like, it seems like a non-trivial task to simulate all of that. Like, it's like, what do you do? Like, you take the person's code and you like fork it or something and do something. And then like the question that enters my mind is like, how do we know the simulation is faithfully simulating the thing? You know, cause it seems like it could be a lot of surface area. So if you could give any thoughts on that, cause like you've obviously thought about it way more than I have. How to simulate these things?
I have thought about it a lot. And you're right, it is not trivial. It is actually pretty hard. It's actually, I think, one of the hardest problems in the world right now. So yeah, a couple things. One, yes, our customers connect to GitHub and we are checking out their entire agent into a sandbox. We have other sandboxes where the agents are actually executing, keeping state. If their agent has sandboxes, we're also spinning up sandboxes to replicate those sandboxes. And there's a whole flow of essentially almost like a smoke testing or alignment flow of we're doing a loop of, well, let's take simulations for the simulations and evals for that. Take production traces from main, rerun them. They should equal main. And so before you're running on any PRs or anything, you're just trying to align with production as it is.
That's really funny. The answer to my question is simulations. I know.
That is actually how you do it, right?
Yeah.
It's like you're trying to align to something, you know what you should be aligned to, so you just keep doing that and keep picking fresh things to try. So that's the first step, is that we call it the world-building phase. And then there's, as far as the faithful simulation, that's what I was talking about earlier. There's a few ways you can eval for faithfulness. One probably more obvious one that you can do after it runs is to have an LLM blindly Decide, look at all, say, say you run 20 simulations. You have an LLM blindly decide, essentially take the original trace, take the new trace. Is it a simulation? Yes or no? If an LLM can correctly guess more than, you know, like would be statistically probable that it is a simulation, that's a pretty good sign that something's wrong. And so there's a few tricks like that that you can use.
And then what about data? So, okay, you connect your GitHub, maybe they have some databases and stuff like that.
Yes.
How does that work?
Good question. So 2 things. This is where the Swiss cheese analogy comes in, which is like the previous run of that agent probably has some good signs of what the underlying databases, etc., had and what you'll need to see for that question. B is that on the fly, let's say you add a new tool for some sort of new database that was never there before, we can simulate and guess what that database would have. Also, we have the ability now for people to connect MCPs to the worldbuilding process. If people want more realistic data, they can either drop in JSON files or whatever, or they can connect an MCP that can be used in the worldbuilding process. That'll actually pull that production data in.
The MCP is meant to impersonate the thing?
It's actually easier than that. Let's say you have some new database. You could connect a read-only MCP just like Let's say you're using Supabase. You could, you know, do a read-only Supabase MCP just so the agent that builds the world for that run can pull the data it needs and understand the schemas from Supabase. And truthfully, the schema can be understood from the codebase, but the data, it allows it to populate that data from production. That is really, we've only really seen that necessary if you're adding like a whole new data source internally that like the agent has just never seen before in its life. It's like if you really want that simulated faithfully, then, then that's something you should do.
Interesting. I think this is really fascinating. Yeah, thank you for sharing this. One question. Okay, so a lot of people are learning evals in this class, and the question always comes up, what are good companies to work for if you are into evals? Do you have any thoughts?
Uh, well, Raindrop, uh, as you just heard, uh, we have a lot of evals. I mean, I think there's a range. I think a lot of our customers are pretty good kinds of people that like care a lot about evals, whether that's Vercel, whether that's hole in is like there's a lot of people in the consumer space. Obviously there's like, you know, bigger companies as well, but I would be remiss not to, not to mention ourselves here. I think ultimately you wanna look for companies where the real-world cost of failures is like high. And I think that it's more than you'd expect actually. Like anything that touches the real world in a meaningful way or a lot of people, probably the cost of failure is pretty high.
Cool. What kind of roles are you hiring for?
We're currently hiring most roles. So it's like sales, head of sales, AEs, and then on the engineering side, we're hiring for FTEs, we're hiring for product engineers, and we're hiring for MLEs. So those are the 3 roles.
All right, cool. I think a lot of people in this class would be interested in checking it out. That's why I asked.
Yes, you can DM me or email me. It's ben@raindrop.ai.
Cool. Probably have some time for some questions. Is there any situation where simulations might be overkill? Like, how do you know you need a simulation? When should you go for it? Should you go for it right away? Should you start something else first? What do you think?
Very good question. I mean, the very first place that you start is just trying it yourself. You have it running locally and just try it. Take the traces, ask an agent what went wrong in the traces, and for it to show you the parts that went wrong. Some way of annotating and seeing those traces. That's honestly why we built Raindrop Workshop. Open source tool, runs locally, streams all the traces directly into it. Do look at your traces locally. It's very, very important. I think it's By far, probably the most important thing you can do. Go through every tool call, understand, and ask an agent like, how could I improve this? How could I optimize this? Like, it really does work. So that's where you should start. I think as soon as you're trying to start running evals, now that you have agents, like, you really kind of need to find some way to simulate it. Now, like, depending on how complex your agent is, like, you may be able to just do this yourself. Like, if you only have like a few tools or like the data sources are pretty straightforward or.
Yeah.
Maybe you have a case where like you can just hook it up into like a read-only version of production and like limit the data it's reading or something like that. Like definitely like try, I think that like you should always try building out your own solutions here because you'll at least understand the problem that we're solving or any other, you know, vendor is solving. But especially if you're doing like classification, there's a lot of things where it's like just run them both in parallel. The problem is that like agents are really expensive. So if you're gonna do a shadow deployment of your agent across all traffic, now you are doubling your costs. And then now you also have to look at those traces still. So like, it's not totally solving the problem, right? And so I would think about it a little bit like simulations are the way to have an action. Like also the iteration loop there is ridiculously slow, right? It's like push to production, monitor for some time, find the things, then like go make another change, push to production. So it's like, I think you want to try to make that feedback loop as tight as possible. On simulations versus experiments and like A/B tests. So Rangetop actually has like, we have an experiments feature that like is probably one of the most heavily used things in our entire app right after issue detection. It's really good. It's really important. Like, I don't wanna like deemphasize the importance of production. It is like production is ultimately like the source of truth of like what is actually happening in your agent, how it's actually behaving. You just like want to do something before you get to production to make sure that like people, uh, aren't, you know, bad things aren't happening.
All right. Amazing. Thank you so much.
