How to Automate AI Evals (Correctly)

This is the first part of a 12-part series, a mini course that we are hosting on AI product engineering, and we're going to kick it off with a bang. Shreya is going to go over how to use LLMs to help you with evals and do so in the right way, because a lot of people are getting stuck on this exact topic.

Thanks, Hamel Husain. Really excited to kick off the series. So I'm Shreya Shankar. I am a computer science professor and I also co-teach, co-created the AI evals course with Hamel Husain. And today I'm gonna talk about how to automate some of the tricky parts of evals effectively. It's not— automating AI evals with AI is not very easy, but I mean, it's not stopping many vendors from trying to sell you automated eval tools. Goal of this talk is not to tear down any particular vendor. In fact, I won't really get into specific vendors in this talk. The goal of this talk is to convince you that you can't just farm out evals to your AI coding agent, but you can still use AI to help you. The goal is to inspire more thoughtful uses of AI when evaluating AI. So what are evals? Eval are in the abstract systematic ways to measure the quality of your application. And building AI products requires evals because the quality of AI outputs has to be good. In fact, according to one of LangSmith's recent annual reports, output quality is the biggest barrier to productionizing agents. And according to lots of Reddit posts out there, figuring out how to evaluate models is actually super hard. It's quite natural that vendors want to automate evals for you. So LangChain, Braintrust, and Arise all has Automated evals tools, and their vision is to basically do evals end-to-end, it seems. So point their tool, which is powered by an LLM, at your agent traces, and then the tool will tell you where your application is broken and fix it for you. So this is a very ambitious vision, and I would love for it to be true. And while these tools can find some errors, they're not a cure-all. And I found that you can't really fully automate evals. And the reason for this is epistemic. What good means for your product is living in your head. It's not in the traces. And if you think about it, if a tool could fully build and fix your product for you, it could do the same for everyone else, all of your competitors, and there would be nothing left to set your product apart from others. So in other words, your judgment is the only differentiating factor of your product. So this begs the question of what can AI do for evals if it can't automate evals end to end? AI tooling, we should think about it as trying to help us developers express and apply our judgment that lives inside our head, apply that judgment faster to the product lifecycle process. So today we're going to go over how to use AI effectively for evals. I'll first talk about the evals framework we teach in our course. First, this lifecycle of evals, and then some common mistakes that we've seen folks making when using AI to automate evals and how to avoid making these mistakes. And I'll go through some demos as well with my own work on how I use AI to do evals. Let's start with a framework. Many of you might already know this, but we teach this 3-step lifecycle for evals: error analysis, measure, and then improve. So the error analysis part is about how to take all of the traces or take the behavior that your agent is exhibiting in traces and figure out what are the failure modes in there. Then the second measurement step is how to measure the prevalence of each failure mode. So if the failure mode is more prevalent, you wanna prioritize fixing that. And then the third step is, okay, actually going deep into the product and improving it. So fixing your product so it doesn't make those mistakes. And this is an iterative lifecycle as your product evolves. So you don't just do all of these three things and then stop. So I'll illustrate some of these eval steps with a tool that we're actually building in my research, an AI writing assistant. So imagine we're building an AI writing assistant and form factor of it is not super necessary right now, but the idea is that the AI can help help users write reports or essays or articles that they need to do. And our users don't wanna use the AI writing and see that it's sloppy. And actually AI slop is a real problem today. So some recent statistics from Graphite show that over half of articles on the internet today sound noticeably AI generated. So we want our assistant to not generate this slop, which is a very vague thing and very hard to do. So how might we think about evals for such an app? First step would be to analyze AI writing for signs of mistakes or signs of bad quality. And analyze is actually the hardest step we found in this whole eval lifecycle because there's no perfect definition of a mistake here. So for example, in writing, I can't perfectly define what slop is. There is this know it when I see it flavor to the problem, and it makes it harder than I found in traditional machine learning, in supervised machine learning, where we had labels for all of our data and we could simply define the failure as the prediction of the model doesn't equal the label. So in this case, we're in a lot of discovery involved. Now, maybe after error analysis, we have found a bunch of different failure modes. The second step is measuring how prevalent these failure modes are. So maybe we have all these failure modes of AI has overused a lot of emojis or it's creating these, overusing some slogans or phrases like, "It's not X, it's just Y." I'm sure you guys have heard of this. What we found in eval is a lot is there's this Pareto principle here. So 80% of issues in the data are often caused by 20% of failure modes. So we really want to find those most common failure modes and prioritize fixing those. So once we've measured how prevalent what the failure modes are, then we can go and improve our AI writing assistant. And there are lots of different ways we can improve it. For fixing a failure mode, for example, we could add an instruction to the prompt, we could switch models, we could fine-tune models. I won't get into all of those details, but hopefully this gives you a sense for what the eval's lifecycle is in the whole process of building a product. And in terms of how we might use AI effectively here, we found that AI is not very good at automating the early parts, the error analysis parts. These are very taste-specific, but the more we go in life cycle, AI can automate and do much more of that for us. So some of this measurement step, some of the hill climbing and improving the prompts or doing prompt optimization, AI could be really helpful at that. So this is hopefully gives you a sense for how to think about where AI fits into your life cycle of doing evals. So now let's talk about some of the common mistakes people make and good ways that we can actually use or infuse AI into our workflows. Let's start with mistake 1, which is opening up a coding agent and simply asking AI to evaluate the application for me. And so often I'll see prompts or people will tell me things like they just opened up Claude Code and then they said, check out my traces. Can you evaluate the app? Here's a pointer to, here's the S3 bucket, or here's the file of all of the outputs. We'll evaluate. And if you look at the reasoning traces, the agent will do something reasonable. The agent will say, okay, let me read the data. Let me understand what we're working with. I promise that this is a real screenshot. I didn't just doctor it. And then 5 minutes later, after thinking, AI will come up with something. Here's what Claude said after reading all the traces. It found some problems. Look fine at a glance. I'm sure many of these are issues we want to fix, but the real issues here are: one, agents can discover some issues, but they will miss all of the case-specific issues. So going back to what I said earlier, anything that lives in your head that's not externalized in the traces or in the prompt, you're never going to find it. The agent is never going to find it in the error analysis. The second point is that agents typically don't find the highest priority issues, and they'll even fabricate what they think is the highest priority. Like in the screenshot here, extremely repetitive voice, the biggest issue. The First of all, extremely repetitive is pretty vague, but then second, it might not be the highest priority issue to fix. There might be other really weird phrases that the AI says or hallucinations the AI makes. Who knows, right? That might actually be more high priority for you to fix. So you can't just rely on what the agent said. And then the third thing, and this is more of an efficiency and a cost standpoint, is that there's no reuse in this workflow. So when you ask AI to find issues in your data, you're asking it to reread the data, come up with new failure modes, do the lifecycle from scratch. And if you want to do it again on a new batch of traces or more traces, there's nothing that's being shared across these different runs, which is very suboptimal from an efficiency as well as accuracy and learning standpoint. What we really want in our workflow is to persist intermediates of the lifecycle. So once we come up with good failure modes and definitions of them, we wanna be able to persist them somewhere in the memory or persist them in code somewhere. We want constant visibility and updates into the process. So that aside, can we ask AI to help us do evals better? One thing I found that's helpful for me is just to stick to my workflow and ask AI to help in very specific parts of the workflow. Instead of trying to automate end to end, maybe AI could help us in the error analysis part. It could help us design human-in-the-loop interfaces or dashboards, for example. I won't get into the AI for measure and improve today, but part of this talk I'll share is the skill that I've developed to help me do AI-assisted error analysis. So I'll walk through it at a high level and then demonstrate it to you in action. So first, the skill instructs the AI agent to look and try to understand the data. So that might be load a sample of the data, which is in some JSON JSON file and then try to come up with different fields or structure or content that's repetitive or shared across these traces. Then the second thing the AI is instructed to do in this skill is to take this understanding of data and design a visual encoding. So think of that as a way to represent all of this data in a UI for a human to review. So this is things like if you find things that vary within a trace, maybe assign different colors to those things that vary. So a, a very simple example of this is the role, if it's an assistant or or user that sent the message. Maybe you can highlight those different dimensions with different colors. So things like this, we're using visual encoding to help the human really understand the data better. And it's not really necessary that you know exactly what this means. This is just a common thing in design that people do to help design good UIs. And the AI is instructed to do this in the skill. Then the meat of the building part is to design a review app to help the human or help myself go through these traces using this visual encoding. So there are 3 views that the AI agent is instructed to make, but this is a trace viewer, so one-by-one trace viewer. The second one is a map view, so you can see a map of all of the traces that need annotation. And then the third is a progress view, so the more that I have labeled or more progress that I've made, more failure modes that I've found, there's a tree map of all of the failures in that progress view. And the fourth part of this workflow is to figure out what samples or what traces to show in the UI. So the agent is instructed in the skill to cluster, to sample representative traces from each cluster. So as a human, I don't have to review every it once. Then the last part is really the human-in-the-loop workflow that goes on with this review app. So the agent has built the app, and then now the human-in-the-loop workflow is to have a human review traces that were selected by the agent, of course. And as the human provides input or labels or annotations in the UI, the agent is instructed to always be monitoring. So you can see the AI agent box in that step 5 here. Always be monitoring, always try to build a taxonomy of failure mode. And then we instruct the agent to go for breadth and depth, which which I'll explain a little bit more. So the idea is go for breadth, try to help the user cover all of the failure modes possible, and then depth, which is for every single failure mode, try to go in and find many examples of that failure mode. I know this is a lot of it as a high level, and now let me get into actually showing you a demo. Okay, so to recap, what I'm trying to do is show you my error analysis skill in action. And here are some traces for my writing assistant. So what you— these are not actually traces. This is just the actual essay. Essays that are finally generated by the LLM. There's a lot going on here, so I don't expect you to be able to read it. Actually, just know that there's a bunch of different essays that are AI generated and we want to analyze them for smells of bad AI writing. So I'll invoke my skill to help me construct this review interface. "Erl2, help me do error analysis on the AI articles." And hopefully as the custom review interface is being created, you'll see what the AI is doing. So I'll start by reading the dataset to understand the structure and the content. Step 1 that I mentioned. Okay, so it's running some shell commands. So you'll see that it's slow because it's actually trying to look at the data here. It'll look at the structure of the data. Great. So there's 33 AI-generated blog articles. That's not very many, but you could imagine that you have a lot more. So now it's going to cluster and select some samples and then build the review interface app. So what— maybe it's useful to talk about why we want to do some clustering and sampling. You can try to review every single AI-generated essay or trace. I found that I get very tired very fast, so it's helpful to get a smarter sample. Unfortunately, sampling with the LLM is really hard. In the skill, the LLM is instructed to find features that could explain how these articles or these traces vary, but of course it's, it's very much a guess. So it's useful to look at samples, but you can't assume that the clustering or that the sample selection is perfect. Okay, so it's preparing some data. I guess this is all the sampling stuff. Yeah, so it's collected 21 samples for the human to with you. Human means me or us. I wrote a server, now building the HTML app. This is the main interface. Little slurred, so I'll write it in one go. So in the first version of this skill, I actually instructed the AI to embed the traces and then cluster the embeddings with k-means. And then I realized that it's not always the best idea to use embeddings. Embeddings may not be the best representation of the trace, or sometimes we might not even be analyzing the traces itself. Like in this case, we're just analyzing the final essay. So the skill is actually very open-ended. It just says find feature that are useful in the data and then cluster. Doesn't specify how to do clustering. And the hope is that the AI agent is smart enough to think about what the data is and reason about what's the best clustering to do there. So I have done— used the skill for error analysis for like data agent traces. Think like data science agents that are helping with data science. And I've used it for this AI-assisted writing. And I don't have a very satisfying answer other than sometimes I'm just very much iterating on the skill. So whenever I see the UI that's created by the agent, sometimes I have feedback for how to make it better. And then I try to encode that back into the skill. So it's definitely very much a work in progress. And I don't know of anyone who has a great error discovery skill. So if you have any pointers to those, please send them to me. So the skill comes from the human. This can be actual user queries. This could also be synthetic queries. It really is dependent on what app you're building. All right. So we're now having promising experiences. Oh, lovely. Okay. So the app has shown up. So hopefully you got a sense for how long it also took. It took, I don't know, a few, 5 minutes, I would say, to come up, cooked for 4 minutes and 42 seconds to build this. So it's actually pretty fast in the grand scheme of things. I promise I didn't pre-prepare anything for you. So now let's make this smaller so that way we can see this UI that it generated. So right off the bat, when I feel like UI generation with these agents, even though we're using some of the best agents, I feel like we have a long ways to go for UI. Don't judge this as like the UI needs to be perfect. It's simply something rough for us to be able to look at data better. I often kind of see, like, it generates a bunch of these labels. I don't know why. Sometimes I get frustrated and I tell my agent, please don't generate all of these pills or these chips. It's really not necessary. But anyway, let's go actually into the error analysis. So I always like to start out with this map view. Now again, someone said I would have done t-SNE instead of PCA, UMAP instead of PCA. Totally use whatever clustering technique that you wanted. This, the AI agent wanted to use this method. But the idea here is that there are 6 different clusters. Clusters, and then some of these orange ones are annotated, and some of them— annotated means select for annotation, maybe not. I don't know why orange is said twice. So again, AI makes mistakes when generating data, so there shouldn't be the same color being used twice. That's a little bit insane. But anyway, so you can get a kind of get a sense for what these clusters are. So maybe there's different topics here that the AI tried to do and so forth. But anyway, so next thing we do is go back to the reviewing. So there's 21 samples that are selected in this, and then I just actually just go through and like actually start trying to think about, okay, say, how is, what are the errors? What are the mistakes in this essay? So this is about container gardening for apartment dwellers. And the agent actually, when generating this article, also had access to the internet and could search for things. So I'm gonna just go through it. I'm not going through it too fast, but I'm, I've done some error analysis on writing before, so I already have an idea of what are AI writing mistakes. One thing is I don't like having colons in each sentence. Or I don't like having colons, and I'll just send that. And so you'll notice here that the way that I give feedback in this app is in situ. I highlight the text and back to the text, I give feedback so that I can externalize it immediately. So that's going on. I keep, can keep going. I want all my sentences to be complete sentences. I want sentences, this to be complete sentences. I'm gonna give one more thing. So I also, good UI principles show progress as it's going. So maybe I'll go to the next one here. And one thing that you should have seen, but for some reason we are not seeing is that the AI agent is instructed to monitor all the feedback that the human is giving. So I don't know why it's not doing that. It's supposed to use the monitor tool. So why aren't you using the monitor tool like I instructed in the skill? So the skill is open source. 'Cause all the annotations get saved to a file. Like somebody asked a question. So yes, all the annotations get saved. I'm using one of the state-of-the-art agents. It's not always perfect. You do need to coax it, cajole it as we go on. I know all of the bad things that you see in this app, I promise they're not hardcoded in the skill. I don't like to this smell of writing, which is, it's not just X, it's Y. I don't like the Y framing and phrasing. Great. All right, so that annotation went through. The monitor event was triggered. 3 annotations just came in. Actually, just one. Yeah, so let me read them and start categorizing it. So you can see the AI agent in the background is simply trying to build a taxonomy of all of these open-ended outputs. So at a meta level, what's happening to the human, I'm just giving my open-ended feedback. I'm not thinking too much. And then all of the tedious stuff around maybe taxonomizing it, categorizing it and so forth, visualizing progress, all of that, let's delegate to the AI agent. Yeah, so taxonomy push, you can see it in the progress view now. So if I go to progress view, I'll see that there are two different failure modes. One is informal fragments and colon interjections. Another one is the contrastive framing. We don't like that. And we could keep going even. So, you know what the great thing was, is whenever I built the first version of the app, I had no idea that these preferences should have been encoded in the system prompt. I thought the AI should simply know, but through doing this error analysis, I'm understanding that these are issues, and then we can think about in the improve step of the lifecycle going back and editing the system. All right, so I'll do one more and then let's go back to the slides. This was the bulk of the demo that, by the way, that I wanted to show you. I just really wanted you to get a sense of what was happening. So what are some takeaways from that experience? I know it was a little bit chaotic, but I love doing live demos. Anyways, point number one is to really build this interactive experience where you annotate data. So it's not just ask your AI assistant to build your application and then you sit and you annotate things in the application. You want to really have this iterative feedback loop where the AI agent is really reacting and responding to you in real time, and you can do that with the skill. The second point is trying to optimize for both breadth and depth, which I didn't have a chance to annotate many examples, but as you're annotating, think about covering, adding as many failure modes as possible in that progress view is very helpful for that. And then as well as for every failure mode, trying to have multiple examples of that failure mode. All right, so the next mistake I wanna talk about is not iterating over data. So many people look at a trace only once when annotating or look at an output only once when annotating. And this is not ideal because research, in fact my research as well, has found that when you go back and you look at traces again, you've already analyzed, you actually end up finding new failure modes. So concretely, why is this the case? When you've seen more traces, you have more traces living in your head, then you can actually do better analysis. When you have different combinations of traces in your head, actually spark failure modes that you discover. So let me actually show you an example of how that materialized for me in this AI writing. But one thing I noticed when I was reviewing essays one by one is that I found new trigger phrases for me that I hadn't seen before. So I already reviewed two. One trigger phrase that I found when reviewing the third one is this overuse of the word matters. So every article for some reason has the word matters in it, including the ones that I already reviewed. So here, let's search for matters. Oh, consistency matters. Once I tell you this matters thing, you're never gonna unsee it. When you see that article on X, you're gonna be like, oh, this break matters. First one matters. Oh, it matters in the high end. So anyways, I didn't like this phrasing of it matters, but this was something I only actually found very deep into my error analysis process when I'd seen the word matters many times. So this kind of goes to show that you really do wanna be going back and looking at things that you've already annotated for new emergence of new failure modes. And one thing I like to do is in this skill, I like to tell my AI agent to look at my annotations and then go back and look at previous traces I've annotated to find instances of those failures. So this is exactly what happened here. The agent suggested new instances of failure modes of traces that already reviewed. So remember on one trace I said I don't like sentence fragments. The agent went and looked at other traces for examples of sentence fragments, traces that are also reviewed, and then I can accept or reject them. So here I wanna accept that. Here's another contrastive framing that I missed. Let's accept that. And maybe there's also more. In my progress view, I usually have an agent suggestions tab, which looks at, again, the things that I've already flagged and then can suggest to me. Yeah, so these are all informal fragments. Typically, I've always found that I just select all and accept these as examples of failures. But then I also note that the agent is not always exhaustive at coming up with these. It's able to flag instances of the failures, but can't flag all of them. Just something to know. Again, it's for accelerating really your own review. So one thing that's interesting here is I don't tell the agent to find me entirely new examples of failure modes. It's not changing the taxonomy. It's not adding to the taxonomy. That's really left to me. I've asked in previous iterations of the skill for the agent to suggest new types of failure modes, but I've always found that some of them I agree, some of them I disagree, and I don't love the experience of just trying to validate the agent's taste. It's much easier for me to say what I think and then have the agent scale that out to applying that to many traces. All right, so what did we learn from this mistake number 2, which is don't try to one-shot discovering all the failures, just only look at your data once. You want to loop over your traces multiple times. So one way to think about this is an outer loop and inner loop framing. So you you have an outer loop of iterating over the data and you have an inner loop of kind of hypothesis iterating through different failure modes within a specific data point itself. And then this other thing that I mentioned of asking your AI agent to take every new annotation you give and try to apply it to previously labeled traces as well as unlabeled traces even. All right, so those were two mistakes. This last mistake that I've seen people make, this is very short, I won't spend too much time on it, is having a uniform accuracy bar for all of your AI applications. Evaluations. So all this means is sometimes people think that they should equally invest in the evals process for all of their different AI products or AI applications in their company. That is not really ideal. How strict should your evals be? This really depends on the context of your application. So for some internal tools, like folks like myself have AI tools for summarizing Slack threads or summarizing data, these things don't need to be perfectly accurate. We don't have to spend too much time investing in evals there. But for anything that's customer user-facing, you have to spend a lot more time having a high accuracy bar there. And One way I try to think about reasoning about what the accuracy requirements of an application are, or eliciting some failure modes, hypothetical failure modes, is to reason about worst-case scenarios upfront. And I actually like to do this with an AI assistant like Claude Code or Codex. I give Codex a sample of all my traces, a description of the application, or even my codebase. And then I ask it to help me reason through if I were to put myself in the user's shoes, like what are the worst things that could possibly happen? So for the writing assistant, maybe it's like fabricating a citation, or it could be leaking some information. That was private. Like, for example, if a journalist is trying to write a story, we don't want all of the information that the journalist has collected often to be in the output of the article. So these kinds of worst-case scenarios can help reason about what kind of evals and guardrails to really look for. So these are just 3 mistakes that I had time to talk about today. Using AI to help with AI evals is a really fast-moving area. As you can see, it's not perfect. We're still working on it a lot, and we're actually in the process of a more thorough, deep deep dive and evaluation of the existing automated eval tools out there. We're benchmarking them on our data, on real data. So this work is actually led by Hamel Husain and Antaripa Saha. I'm really excited for it to come out. Interestingly, we found that first, general-purpose coding agents like Claude Code and Codex, what I showed you, often end up being more exhaustive and good at finding failure modes than some of these dedicated discovery platforms. So that was very surprising to me. I guess the agents are pretty good. And this other thing is that no, there's no such tool or no such evals tool can find all of your failure modes with very good precision. So stay tuned for this research to come out.

Yeah, one thing I'll say about it is it might be surprising that these— there's like automated evals tools from vendors may not give you that much lift over Codex or Claude. And the reason that is is because at the end of the day, it's someone's prompt. Really what these eval tools are is someone's prompt, maybe a little bit of a harness, but you can likely inject your own domain expertise and prompt it yourself. And then also you can also customize your coding agent to your data. And with that, like you can really close the gap. That's like the kind of some intuition behind this, but we'll be sharing a lot more details as Shreya mentioned.

All right. So to summarize my talk today, we discussed the analyze, measure, improve lifecycle for evals. We talked about 3 mistakes that's very easy to make when using AI to automate evals. So don't ask your AI to do evals. Split this, your workflow into the steps. Engage in an interactive workflow with your AI. Second, don't just do one pass over your traces. Loop over them to surface new failure modes. Third, don't just have a universal accuracy, a uniform accuracy bar across all of your applications. Really reason about what applications are customer-facing, some worst-case scenarios. Use those to work backwards into what evals and guardrails you wanna have. The scale that I showed you is open source, so you could check it out. Really in flux. I always edit it and keep updating it. And last but Last but not least, I'm teaching a new version of the e-vales course this fall. Hamel Husain and I are totally revamping it. Super excited for that. So you can sign up. I'm also hiring undergrads, master's, PhDs for my research lab. So if you're interested in doing research, I can be reached at this email address. This slide really captures how I think about using AI for e-vales, which is figure out how to express your taste that's hidden in your brain and your judgment into your product development workflow. And this is pretty hard to do. So, and looking at the data is the best way to do that because it will trigger some concerns that you have about your application and then you can externalize it to the AI. I don't know of a way for AI to probe you 20-question style for how to externalize taste very good. So there will be some human element in evals always. Okay. We'll figure it out later. So thanks everyone for engaging and hope to see you next time.

Thank you.