Five hard earned lessons about Evals — Ankur Goyal, Braintrust

Ankur Goyal, Google Developer Relations. Let's talk about some of the interesting things we've learned over time. So the first thing is I think it's super important for you to understand and define whether evals are actually providing value for your organization or not. Um, and I tried to come up with 3 signs that you should look for, um, that, that are good. Uh, so the first is, um, if a new model comes out, uh, you should be prepared, um, uh, via your evals to be able to launch an update to your product within 24 hours that incorporates the new model. Uh, Sarah from Notion, um, she talked yesterday. She talked about this, um, specifically. But, um, for the past several model releases, every time something comes out, Notion's able to incorporate the new model within 24 hours. And I think that's a really good sign of success. If you can't do that, then it means that you have some work to do on your evals. Another sign of success is if a user complains about something, do you have a very clear and straightforward path to take their complaint and add it into your evals? If you do, then you have a shot at actually incorporating user feedback, pulling it into your evals, and ultimately doing it better. If you don't, then you're going to lose a lot of valuable information into the ether. So again, I think this is a really important kind of threshold or milestone to hit. And the last one, which I'm actually going to talk about a little bit more throughout the presentation, is you should really start using evals to play offense and understand which use cases you can solve and how well you can solve them before you actually ship things. Not like unit tests, which allow you to just test for regressions. And so if you— if you really adopt evals, then I think before you launch a new product, you have a really good idea of how well the product might work, given what your evals say. The second lesson is that great evals, they have to be engineered. They don't just come for free. With synthetic datasets and random LLM-as-a-judge scores that you read about online. And I think there's maybe two ways of thinking about this. There's no dataset that is perfectly aligned with reality. I think in the cases that there are, there's like basically nothing to do and the use cases already work, which there are a few that, that are kind of like that, like solving competition math problems, for example. But for most real-world use cases, any dataset that you can come up with ahead of time is not going to represent what users are actually experiencing, and I think, um, the best datasets are those that you can continuously reconcile, um, as you actually experience what happens in reality, and doing that well requires quite a bit of engineering. Um, of course, Braintrust can help you with that, but I think the— the point is you have to think about, uh, a dataset as an engineering problem, not just something that's given to you. And the same is true with scorers. I think, um, a lot of people we talk to ask, "Hey, what scorers does Braintrust come with, and how can we use those so that we don't need to think about scoring?" And we actually have a really powerful open-source library called AutoEvals, but it's very open-source and flexible for a reason, which is that every company that we work with that's sufficiently advanced is writing their own scoring functions, and modifying them constantly, and I think one way to think about scores is they're like a spec or like a PRD for your AI application, and if you think about them that way, one, it actually justifies making an investment in scoring beyond just using something off the shelf, and two, hopefully it's fairly obvious that if you just use, you know, an open source or generic scorer, that's a spec for someone else's project, not yours. [Speaker:ANKUR GOYAL] There's been a real shift towards context in prompts that's not just the system prompt that you write. And I actually think that just traditional prompt engineering— people say this in different ways, but I think traditional prompt engineering is evolving quite a bit. And it's very important to think about context, not just a prompt. So this is an example of what kind of a modern prompt looks like for an agent. Usually you have a system prompt and then a for loop, which, you know, runs LLM calls, issues tool calls, incorporates the tool calls into the prompt, and then iterates and iterates. And I actually took a few trajectories from agents that we see in the wild and summarized these numbers. And as you can see, a vast majority of the tokens in the average prompt are not from the system prompt. And so yes, it's very important to, write a good system prompt and continue to improve it. But if you're not very precise about how you define tools and how you define their outputs, then you're leaving a lot on the table. And I think one of the most important things we've learned together with some customers is that you can't just take tools as a reflection of your APIs or your product as it exists today. You have to think about tools in terms of what the LLM wants to see and how you can use, you know, exactly what you present to the LLM to make it work really well. And I think that in most projects, it's actually very disruptive when you write good tools. It's not something that's just like an API layer on top of the stuff that you already have. And the same is true with their outputs. There's one example that we worked on recently for an internal project where shifting the output of a tool from JSON to YAML actually made a significant difference. And I know that's a little bit of a meme in the AI universe, but it's just so much more token efficient and easy for an LLM to look at YAML-shaped data while doing analysis than extremely verbose JSON. Now if you're writing code and you're plugging something into, you know, a charting library, it makes no difference because to JavaScript, YAML and JSON are both structured data, but to an LLM they're very different. And so I think you have to be very, very thoughtful about, you know, how you actually construct the definition of a tool and how you construct its output for the LLM to maximally benefit from it. So I think one of the most important things we've learned, and actually I would credit some of the folks at Replit for really pioneering this pattern. Um, but, you know, every time a new model comes out, uh, everything might change, um, and I think you need to engineer your product, engineer your team, um, engineer your, you know, mindset so that when a new model comes out, if it changes everything for you, you can jump on that opportunity and, and ship something that maybe wasn't possible before. Um, and I'm gonna show you some numbers, uh, for a product, uh, feature that we're actually launching, and I'm gonna show you a little bit of it today, But we've had an eval for a while that tells us how well this feature might work. And we run it every few months. And you can see, you know, it wasn't that long ago that GPT-4o was the best model out there. But things have changed. And, you know, progressively GPT-4.1 did a little bit better. 3.7 Sonnet is much better. And 4.0 Sonnet is actually even more remarkably better. And what that's meant for us is that this feature that, you know, at 10% would really not be viable for our users to use suddenly becomes viable. And so, you know, Cloud 4 Sonnet actually came out 2 weeks ago, and we're shipping the first version of this feature today, which is just 2 weeks later. But we were able to jump on that opportunity because we ran this eval, we were ready to do it, and we saw that, okay, great, we've actually finally crossed this threshold. Um, so everyone that I personally work with or talk to, I encourage to create evals that are very, very ambitious and, um, likely not, uh, viable with today's models and construct them in a way that when a new model comes out, you can just plug the new model in and try it. Um, in Braintrust, we have this tool called the Braintrust Proxy. Um, there's a lot of similar tools. You could use ours or you could use something else. But really the point is that you don't need to change any code to work across model providers. And so, you know, Google just launched the newest version of Gemini. Actually, Gemini 2.5 Pro 0520 scores 1% on this benchmark, so we didn't even put it on here. But maybe the thing they launched today actually does a lot better. We can find out, you know, with just a few keystrokes, maybe right after this talk. Um, and the last thing is it's super important if you, uh, think about, um, optimizing your prompts to optimize the entire system. Um, so that means, uh, thinking holistically about your, uh, uh, AI system as the data that you use for your evals, the task, which is, you know, the prompt, the agentic system, tools, et cetera. And the scoring functions. And every time you think about making your app better, you need to think about improving this overall system. We actually ran a benchmark, which is the same benchmark that I showed previously. It auto-optimizes prompts using an LLM. And we ran it once by just giving it the prompt and saying, like, hey, please optimize the prompt. and a second time giving it the prompt, the dataset, and the scores and said, "Please optimize this whole system." Um, and you can see there's a very dramatic difference. So again, um, it— something goes from unviable to viable, um, but it's just super important to optimize the entire system, not just the prompt. And actually, uh, this is, uh, a new product feature that we are starting to launch today. If you're a Braintrust user, you can go to the feature flag section of Braintrust and turn on a new feature flag called Loop. Um, and, uh, Loop is this amazing, cool new feature that actually auto-optimizes your evals directly within Braintrust. Uh, so, uh, you can work in our playground and, um, give it, uh, you know, a prompt, uh, a dataset, um, and some scores, and it can actually create prompts, datasets, and scores to, and just, you know, work with it. The kinds of things that we've seen work really well are optimize this prompt, or what am I missing from this dataset that would be really good to test for this use case? Why is my score so low? Or why is my score so high? Can you please help me write a score that is, you know, harsher than the one that I have right now? You can also try it out with different models. So as you could see from this, we've definitely seen the best performance with Claude 4 Sonnet, and Claude 4 Opus performs a couple of percentage points better. But we encourage you to try it out with different models. You can use O3, you can use O4 Mini, you can use Gemini. Maybe you're building your own LLM or fine-tuned model. You can try that as well. And yeah, we're very excited. For this, I think I'm gonna talk about this a little bit later, and I'm happy to do it with some Q&A as well, but I actually, I really think that the workflow around evals is going to dramatically change now that LLMs are capable of looking at prompts and looking at data and actually making, you know, constructive improvements automatically. A lot of the manual labor that went into iterating with evals doesn't need to be there anymore. So it's really exciting. We're excited to ship this and start to get some feedback. So just to recap, 5 lessons that I think are really important. Effective evals speak for themselves. It's important to understand whether you've kind of reached a point of eval competence in your organization or not. It's okay if you haven't. It's not easy. But it's important to be honest about that and work towards it. When you're working on evals, it's very important to engineer the entire system. So don't just think about the prompt. Don't just think about improving the prompt. Please don't just use synthetic data or Hugging Face datasets. I know they're awesome, but please use more than just that. Please don't use off-the-shelf scores only. Write your own. Think very deliberately about how you can craft the spec of what you're working on into your scoring functions. Think very carefully about context. And I think in particular, what helps me personally is to think about writing tools like I would think about writing a prompt. It's my opportunity to communicate with an LLM and set it up for success. And how I define the API interface of the tool and I define its output has a very dramatic impact on that. Make sure that you're ready for new models to come out and to just change everything. Um, so if a— if a new model comes out, you want to be prepared to know that immediately, ideally the day that it comes out, um, and also be prepared to, like, rip out everything and replace it with a fundamentally new architecture that takes advantage of that new model, and I think part of that is obviously having the right evals. Part of it is engineering your product in a way that actually allows you to do that, and then finally, when you think about optimizing or improving your eval performance, you have to think about optimizing the whole system, the data and how you get that data, the task itself, which, you know, the prompt, tools, et cetera, and the scoring functions. And with that, we have some time for Q&A. Yeah, there's two microphones up here, one on the left side, one on the right side. Feel free to stand up and ask your questions.

Hi, this is Jyoti. One of your slides said take feedback and turn it into an eval. Are you concerned about overfitting evals at that point where every feedback then turns into an eval?

Oh, that's a great question. Also nice to see you. So the question was, one of the slides was about taking feedback from real data and adding it to a dataset and incorporating it in an eval. Are you worried about overfitting? And I think the answer is, I'm actually way more worried about overfitting to the dataset without the user's feedback than I am to adjusting the fit to incorporate the user's feedback. The most important thing about a dataset is not the state of the dataset at any point in time. It is how well you are equipped to reconcile the dataset with the reality that you want. Um, and I actually think one of the things that we discourage, uh, in the product— and some people complain to us about this, I get it, uh, if you're one of those people— um, but we don't automatically take user feedback and add it to datasets right now. We actually want a human who has some taste and— maybe can build some intuition about the problem to find the data points from users that are interesting and add them to the dataset. And I think that is your opportunity as a user to apply some judgment about, oh, OK, this user is trying to do something that should obviously work. It's really sad that it doesn't work in my product. Let me add it to the dataset so I can make sure it does. Excuse me. You had a slide, I think, in the tool descriptions about— with some percentages on it. Yeah, this one. What is that? Yeah, so we took a few agents, like we, you know, have a lot of traces, and we analyzed the relative number of tokens for different message types. So the system prompt is one message type. Tool definitions are, you know, the spec of what tools the model can call. User and assistant are, tokens from user and assistant, just text interactions. And then tool responses are tokens from the, you know, the tool generates itself. Oh, and this is the percentage of tokens? Correct. And this is the relative percentage of those tokens. Yeah. Yeah. Yeah. So the point that we're trying to make here is that I think in modern agentic systems, tools actually, like, very, very significantly dominate the token budget of the LLM. And I think that it's very important to think about how you define the definition of tools and how you define their outputs so that you engineer the LLM for success, not just sort of take your GraphQL API and give it as a bunch of tool calls to the LLM.

First off, that point about the thumbs down is, Such a good point. I'm working with the government, and people don't like the answer they got, for example, about taxes, and they give it a thumbs down.

Yeah.

Right? So adding that human aspect is a really good idea. We actually even added a little thing that said, the answer's right, but I just don't like it.

That's awesome.

But my question is about your point that the new model changes everything. We've updated our models several times and use Claude and OpenAI, and we haven't found huge differences other than recently someone really cheap wanted to use 4.1 mini and like it seemed to ignore every— and I swear it ignored the system prompt completely. Yeah, but what kind of things? When you say it changes everything, can you tell me a little more about what kind of changes you're seeing?

For sure. I think the use case that we just shipped with Loop is a really good example of that. So this is a very ambitious agent. It's looking at prompts and datasets and scores and automatically optimizing the prompts based on the datasets and scores. And this is something that, you know, we wrote a benchmark for a while ago and we ran with every consecutive model launch and the the numbers looked more like what you see for GPT-4o for a very long time. This isn't true for every benchmark. So as part of this exercise, we actually have a bunch of evals that Loop optimizes. That's our eval set. And there's some evals like classifying, taking movie quotes and figuring out what movie they're coming from that have worked really well since GPT-3.5. And so there's certain use cases where it just doesn't matter. There are other use cases where, um, they're so ambitious that they just don't work today, and I think you want to create evals, uh, so that if there's something ambitious that you want to do in the future, you are very well prepared when a new model comes out to just push a button and find that out. Okay.

Thank you.