I Ranked Every AI Eval Concept From Worst to Best

Hey, how's it going? If you've been watching this channel, you know that I focus on AI evals. I've been working on evals for over 2 years, and I've worked with over 50 companies improving their AI products with evals. And I see the same mistake over and over again. So what I'm going to do is this fun exercise where I have all of these eval concepts, and we're going to rank them from S tier to F. So let's get right into it. First thing, generic metrics. What are generic metrics? These are things like helpfulness score, conciseness score. They're very generic. They usually come with eval tools and they involve prompts written by someone else. I'm going to put this in the F tier because this is a giant waste of time. If you have generic metrics like this on your dashboard, they're probably not telling you anything and they're usually not useful at all. Okay, the second thing, error analysis. What is error analysis? Error analysis is a process that helps you look at your data. It's like a structured approach to looking at your data quickly so that you can find what's broken in your application. I'm going to put this on the S tier. If you're not doing this, you're doing evals wrong and you need to be looking at data. We'll talk more about that later. Okay, here's the next one, writing evals before building anything. So what I mean by this is before you start writing evals, you sit down in a room or maybe a dark corner and you just brainstorm All the things that could possibly go wrong with your application, and then you write a whole bunch of evals for that and you start there. I would say I'm going to put that in the C tier. You don't want to do this because there's an infinite surface area of things that can go wrong in your AI product. You're just going to waste a lot of time if you just come at it with a top-down approach like that. It's not the worst thing in the world because at least you're doing something. But it's a bad way to prioritize your time. You should be looking at data first and seeing what's actually broken and let that help you prioritize what evals to write because evals aren't cheap. Custom annotation tool. So this custom annotation tool, basically, like, you should be writing a custom annotation tool to look at, help you look at your data. So I'm gonna put it in the S tier. And what I mean by this is like use Claude, Codex, whatever. And create a custom annotation app that removes all the friction from looking at your traces and make it really good. So like, let's say you have an AI system that helps you write emails, render it like an email. It's way better than spreadsheets, and there's just a lot of alpha there in creating your own custom annotation tool. Okay, the next one. Let's say you have a suite of evals and your evals pass 100%. Like your whole suite of evals, they're all 100% passing against all your test cases. Is that a good thing? I would say that's D tier. If your evals are always passing all the time on all your test cases, they're not telling you anything at all. They're just passing all the time. You should think of evals a little bit different than your unit tests in software. Like you don't want them to always be passing. You should always have some hard to reach evals. that are challenging, that you're working on, so that you can use those evals to help improve your application. Reading traces yourself. This is again S-tier. This goes back to look at the data. There's nothing more valuable than looking at the data. Off-the-shelf LLM judges. What does that mean? If you go to a popular evals vendor, a lot of times they'll have an off-the-shelf LLM judge with a prompt someone has written for you. I would say this is F tier. This is the same thing as generic metrics. Don't do this. Don't use those things. I challenge you to go read those prompts. Those prompts rarely make any sense and they're super generic, usually not good for your product. Okay, prompt optimization without evals. This is probably also F tier. Like, what are you even optimizing against if you don't have evals? I see this all the time where people just ask Claude to improve their product and give like very vague description of what good looks like. It's usually a bad idea. Your agent will end up producing a lot of slop. Don't do this. Think about evals a little bit before you jump into prompt optimization. Look at your data before you jump into prompt optimization and think about what you're trying to optimize against. Okay, binary pass/fail metrics. I'm gonna put this in A tier. So what I mean by that is If you're building any kind of eval, especially LLM as a judge or a classifier, you want the output to be binary rather than Likert scale. Binary is just going to cut through a lot of noise and help you clarify your decision-making in terms of what is good and what is bad. If you have a Likert scale like 1 to 5 or even a score from 0 to 100, oftentimes you're going to hide a lot of complexity and kind of kick the can down the road in terms of decision-making. You can hide those, hide that in those middle values, and those aren't really that interpretable. So I would discourage people from doing that generally unless you have a good reason to. And I'm going to go through these really fast, by the way. I have an evals guide that talks about each of these things in a lot of detail. You can find it if you go to bit.ly/evals-guide. Okay, so the next thing is public benchmarks as a substitute for product evals. Okay, so we We know that there's public benchmarks out there, TerminalBench, SuiBench, you name it. Should you just use those as benchmarks somehow as your evals? I would say absolutely not. Those benchmarks may help you choose a model, but they usually don't correlate that well to your product-specific failures. You need to create evals for your product. The next one, only devs annotate data. I'm going to put this in D tier. So unless you're building a coding agent, your developers should not be the ones annotating data. You should have the domain expert be the ones that are annotating data. Usually that's someone like a product manager, or if it's some— if it's an expert assistant like a medical AI assistant, then you would need like a doctor or somebody with enough expertise to be reviewing the data and not a developer. Designing for verification. This one just refers to, hey, like, design your application such that it's easy to verify, meaning show your experts intermediate outputs. Don't just show them one shot the answer. I'll put it in B. It's kind of advanced. It's not necessary, but it's a really good idea. 1 to 5 ratings. We already talked about this. Don't use Likert scales. I'm gonna go ahead and put this in F because I hate it. I think it Wastes a lot of time, and, and it just makes you spin your wheels as an engineering team or a product team in general to have non-binary outcomes from your evals. Okay, one judge for everything. I see a lot of teams just like push everything into one judge. They have like one giant prompt for the judge of like every thinkable mistake. And honestly, the judge in these situations are scoped way too I actually haven't put that in the F tier. I think that's really bad. Your likelihood of success is very low if you just have one judge for everything. Codebase checks, S tier. You should always try to have cheap evals wherever possible. So LLM as judge is the most expensive kind of eval you can have. A classifier is even cheaper, something like Jev. Even cheaper than that is code. So Try to use the cheapest eval machinery you can. Obviously, you want to make sure that your eval agrees with human labels. You want to validate that. But within that, use the right tool for the job and try to see if you can be clever and use cheaper methods because it pays off. Okay, evals based only on brainstorm failures. This again is F tier. We kind of talked about this, like don't sit there and brainstorm failures and use those to drive evals. Do error analysis. At least you're doing something. You know, you should look at your data first. So let's make that C tier. Hey, it's really the same as writing evals before building. Generic helpfulness and coherence scores. We already talked about this. That goes in the F tier. Okay, benevolent dictator. So if you are annotating your data by committee, that is super painful and super costly. Where possible, try to appoint a benevolent dictator. I'm gonna put it in A tier. Benevolent dictator is someone that you trust to annotate your data, someone that can make the final call. And hopefully that's the person— you only need one person for that specific feature, product, or geography to be able to annotate that data. It doesn't have to be one person for everything, but the fewer people that you can have annotating data, the better off you are. Okay, uh, picking an eval vendor first. I'm going to put that in F tier. So if the first thing you do when trying to solve for evals is you go pick a vendor, you're in trouble. Don't do that. It's not about the tools. Focus on the process. Look at the data. Write some minimal evals. Understand what's happening. Don't go straight for the tools. If you go straight for the tools, you're likely going to be led astray. Don't do that. Focus on the process. Okay. ROUGE and BERT score for answer quality. These metrics don't even make sense a lot of times for answer quality. You don't want some kind of string similarity score like ROUGE for AI outputs. And you don't use generic metrics in general, just don't use that. Create product-specific metrics for your failures. Running always passing evals less often. This is a really good thing to keep in mind. I'm gonna put it in A tier. So evals are not free. It costs money a lot of times to run evals, especially if they're not codebase evals, especially if they're LLM as a judge. You want to weigh the costs and benefits of the eval. And so if it's always passing, that's one sign that the eval is becoming less useful to you and maybe consider retiring it. Okay, the next one, immediately writing an eval for every failure. I'm going to put this in the D tier. I'll put this in the F tier actually. So if you immediately write an eval for every failure you see, It's probably a bad idea because, again, evals aren't free. You have to weigh the costs and benefits. So if the eval— if the failure is really bad and you don't know how to fix it, that's an amazing situation in which to write an eval because that eval is probably going to provide you with value. It's not only useful to catch that error, but you're going to have to iterate a lot over a period of time to fix that error. And that eval is going to help you because it's going to give you a goal to hill climb against. Okay, next one, validating judges against human labels. I'm going to put that in S tier. So most people write an LLM judge, they just write a prompt and they say, this is my judge. But how do you trust that judge? The only way to trust a judge is to check it with human labels. And so validating with human labels is S tier. Okay, off-the-shelf guardrails. Off-the-shelf guardrails, F tier. Very bad idea for the same reasons that generic metrics are a bad idea. You don't want to just blindly use someone else's prompts or use someone else's classifier. Usually it's a bad idea. They're not tuned to your product. Don't do this. Most of the time I see off-the-shelf guardrails or just someone else's prompt. Deciding on a rubric before reading data. I'm going to call it C tier again. This is the idea that you are just sitting there brainstorming, ruminating about what can go wrong about your AI product. Don't do this. Look at data. Look at data first. It should be the first thing you do is look at data. Next, domain experts writing prompts. I'm going to go ahead and put that in the S tier. So a lot of people outsource everything that has to do with AI to developers. Don't do this. The whole point of AI is to have the domain expert closer to the product so you don't have this separation of concerns. You should have your domain expert— let's say it's a legal assistant— you should have your lawyer writing the prompt. And then finally, the last one is letting AI do all your evals. I'm going to put this in the F tier because this is how you can get led astray. A lot of people just use Claude Code or Codex to go find all the problems. That can only catch some issues, but I guarantee you it won't catch all of your issues and it won't catch the most important ones, the tricky ones that take context, taste, and judgment to understand. I went through that really fast. If you want to learn more about evals, I made this detailed guide that walks through everything you need to know about evals depending on what stage you're at with evals. So maybe you don't know anything about evals. Or maybe you have evals, but you don't really know what to test. Or maybe you don't trust your eval scores. Or maybe your product feels too difficult to eval. Or you're finding that it's too expensive. Whatever the case may be, I have a detailed guide for you that'll walk you through what you need to know. You can find it at bit.ly/evals-guide. Click the link and we'll send it to you.