7 Most Common AI Eval Mistakes (From Teaching 5K+ Engineers & PMs)
In this 30‑minute tutorial, Hamel Husain draws on his experience teaching over 5,000 engineers and product managers and consulting with 50+ companies to expose the seven most frequent pitfalls that teams encounter when building AI evaluation pipelines. He argues that reliable evals start with a deep look at real interaction data, then move to purpose‑specific, binary metrics that are directly tied to observed failure modes. He stresses the necessity of validating any LLM‑based judge against a human‑labeled train/dev/test split and measuring precision/recall, not just accuracy. The talk also covers product design choices that make evaluation feasible, the dangers of Likert‑style scoring, the importance of domain‑expert annotation, and the limits of fully automated eval tools. Throughout, Husain illustrates each mistake with concrete examples—a leasing‑assistant conversation, a data‑agent dashboard, and a comparative study of leading eval vendors—while offering practical flashcards, a free FAQ, and links to deeper resources. The overall message is a step‑by‑step framework for constructing trustworthy, actionable AI evals that align with product goals and business decisions.
Chapters
I've taught evals to more than 5,000 engineers and PMs and worked with over 50 companies on their evals. And I've seen a lot of the same mistakes which lead to teams wasting their time posting unreliable evals or chasing down the wrong problems. So in this video, I'm going to walk through the 7 most common mistakes people make with evals so you can build evals that you actually trust. Number 1 is creating evals before you look at any data. So if you create evals before looking at any data, you're going to waste a lot of time. If you start with just brain brainstorming anything that can go wrong, what happens is the space of what can go wrong is infinite. And so you're going to end up wasting tons of time and you're going to have a dashboard full of metrics that are not going to point you in the direction of what you need to work on. Let me show you an example of what I mean. Here is a simplified conversation from an AI leasing assistant we've worked with. The purpose of this AI leasing assistant is to help property managers handle conversations with prospective tenants. So this is one real conversation that we found when looking at the data. After a bunch of back and forth, the prospect eventually said, hey, this is out of my budget. Thank you for your business. And then the leasing assistant went ahead and said, you're welcome. If your situation changes or have any other questions in the future, feel free to reach out. Have a great day. To some people, this might sound great. If you point AI at a conversation like this, it's going to say everything went well because it looks like the customer is happy. But remember, this application is a sales assistant. It's supposed to nurture the prospective tenant. And so in this particular situation, it could have done a better job at looking for other properties owned by the same company that would have fit that person's needs. In this specific situation, the AI didn't do anything in the background to try to handle this objection, and it just said, okay, thank you. And so this is an example why you need to look at data so that you can remind yourself about what good looks like. And ground the evals that you choose to write in actual problems that are coming up. The second mistake people make is using generic metrics off the shelf. So if you go to any evals vendor or any evals tool out there, they'll serve you up a menu of generic metrics that you can just put on a dashboard. And it feels like progress, but there is no better way to waste your time. What you'll end up with is a dashboard that looks like this: helpfulness, truthfulness, conciseness score, all on a scale of 1 to 5. And it looks good, but no Nobody knows what it means. What does conciseness 4.9 mean or helpfulness 4.2 mean? It's not actionable and it's so generic that it's a waste of everyone's time. We don't know what a 3.7 versus a 4.3 means. You don't want to do this. You want to start with looking at your data. An example of a better metric would have been, did we handle sales objection when a qualifying opportunity was presented? That is specific. That is something that's going wrong with the application, and that's something you want to measure rather than these generic topics. Just to expand on this topic further, I have this flashcard that helps drive home this point. So you don't want generic metrics like RUE, BLUE, faithfulness, helpfulness, and you want these specific metrics like a calendar scheduling failure, interrupted conversation flow, a widget not rendering, maybe the email recipient is incorrect, or maybe you have failed to escalate to humans. Can see how these errors are specific and not generic like that helpfulness score. And so what you want to think about is you want to make sure your eval metrics correlate to something you have observed. There's some data that you observed, or at the very least, it's something you have been able to reproduce. If it's some error that you're really worried about happening, you should try really hard to reproduce it before creating the eval. You also want your eval metrics to relate to a non-trivial issue. Don't waste your time creating evals for things that can be fixed Super easily, as evals are not free. You also want your evals to be scoped to a specific failure. You don't want a catch-all eval. If you have a catch-all eval with a giant prompt of anything that can go wrong, that's very similar to a generic metric, and ultimately it's not that helpful. You also want to try to make sure your evals have a binary outcome, pass or fail, instead of some kind of score, because a score hides a lot of complexity in it, and it becomes exponentially more expensive to align your team on, on what a 1, 2, 3, 4, or 5 means. And finally, you want your evals to be verifiable. You want to use human labels so that you can measure your evals against a ground truth and build confidence around how good that eval is. And that brings us to our 3rd mistake: not validating your LLM judge or eval against human labels. The last situation you want to be in is where you are challenged to explain why your LLM as a judge can be trusted or why your eval can be trusted. You don't want to just write a prompt and assume that the eval is good. That would be the same thing as just praying that the eval is good. Instead, what you want to do is have human labels, and you want to test to make sure those human labels align with your LM as a judge or classifier. And so I have this flashcard here on the screen, and basically what you want to do is segment your data. You want around 20% of your data to be train, 40% to be dev, and 40% to be test. And essentially what you want to do is when you're writing your LM as a judge prompt, you want to source examples from that train set. And then that dev set, you want to tune the LM as a judge to agree with the dev set as much as possible. So basically you want to hill climb against that dev set, and that test set you want to hold out or hide from yourself to the very end just to make sure you didn't overfit to the dev set. You want to make sure that any LLM judge or classifier you create can generalize outside of the data that you or the LLM or the classifier has seen. Now you shouldn't just measure accuracy, you should measure precision and recall, and that's what TPR and TNR is. I won't go into the details of those calculations, but the most important thing to take away here is that if you have an LLM judge, you need to align it with human labels. You need to measure it against human labels. Beyond these mistakes, there's a lot of questions that come up with evals. So I've put together this free AI evals FAQ with the most common questions that we get during our evals course office hours. We're always adding to it, and I'll leave a link in the description below. The 4th mistake is not designing your product for success with evals in the first place. Let me show you an example. So what I have on my screen is a data agent. The purpose of the data agent is to answer business questions like what was net revenue for product A last quarter? Now, a lot of times people design such a thing as a chatbot that just gives you the answer. And the problem is, how are you supposed to evaluate this? The only way to evaluate something like this would be to redo all of the work, and that would become extremely expensive to evaluate. There's no way that you could write evals or look at this data without it consuming a lot of time. And what you want to do is design your product differently such that it sets you up for success with evals, but also is a better user experience. Anytime you have an AI product that caters to an expert user like this data agent, what you want to do is design for that persona in mind. So instead of something that looks like this when I'm showing on the screen, you want to design something that maybe looks like this. You ask a question, but it shows you where the answer is coming from, and it shows you the intermediate calculations. It shows you correlated numbers, so on and so forth. And it might also show you a lot of detail around how it was computed and give you the chance to walk through step by step how that number was computed and even change the way it's computed and give you a bunch of trust signals such as how was this metric computed previously? Are there any related reports? Where this number is consistent with what was previously reported, etc. If you design your product like this, you have a much better chance of being successful with evals. And it's not just constrained to data agents. There are other examples in this blog post. I talk about a curriculum builder for schools where instead of just an app that just spits out a lesson plan, what you want is an app that walks the teacher through the changes from an existing trusted lesson plan and allows them to accept or reject those changes. And similarly, I have another example in the blog post, which I won't go through right now, about a medical use case. I'll put a link in the description so you can go through this more carefully. The next most common mistake at number 5 is using a Likert scale or a 1 to 5 scale with an LLM judge. And I've shared a flashcard on the screen to help illustrate the point. It's really common when you're building an LLM judge to think you want a score, like a 1 to 5 score or a 0 to 100 score, because in life, a lot of times there are shades of gray. There's no black and white. Something is kind of good but not great. And you don't want to carry that intuition over into the way you construct evals. You want to create evals that mirrors the way that you are going to make product and business decisions. What do you want to think about is, is this good enough to ship? Yes or no? And be very clear about what is good and what is not. And it's very likely in your business that you have a line of what is good enough and what is not good enough to ship. If you are constructing evals that output a 1 to 5 score, you're going to be trapped in massive complexity. You're going to have to have a rubric that specifies what does a 1, 2, 3, 4, 5 mean. And you're going to have to spend a lot of time aligning humans on exactly what that means. And it's going to slow down the process exponentially. You don't want to do this. Another thing you want to do is scope your evals so that they are focused on a single error. When you scope your evals like this, it becomes much easier to create a binary score. So in this diagram, we're showing you how to do things correctly. Again, you don't want this Likert score. What you want to do is have binary pass/fail outcomes and scope your evals to a very specific error like is polite or is there a scheduling error or was there a human handoff error and have those be pass or fail. Those are more actionable and going to be way easier and less expensive to manage. Having a 1 to 5 Likert scale will just trap you in complexity and result in meetings that go on for hours. You don't want to do this. Number 6 is having the wrong people review the data. So it's very tempting to outsource evals to your developers. After all, your developers are the ones who are making the AI application. However, your developers are not the domain expert in most situations unless you're building a coding application or a coding agent. The people that should be reviewing the data are the people who understand the data. So if it's a legal assistant, it should be a lawyer. If it's a medical assistant, it should be a doctor or a person with enough medical expertise to understand whether or not the outcome is good or not. To do this well, you want to create your own annotation apps. You can use Claude Code, Codex, or your favorite coding agents to create a customized annotation app that makes it easy for that expert to review the data. This is an example of a custom annotation app for that apartment leasing AI assistant that we discussed earlier. This is customized to the data and allows an expert, which is a product manager working on this application, to review the data. It is a really good idea to do this so your domain experts are the ones who are annotating and reviewing data, not the developers. For number 7, the mistake is relying too much on automated approaches or automated evals. So I have a meme here on the screen, and people literally do this all the time, is just ask a coding agent to find all the errors. Now, a coding agent can find some errors, but it's not going to find all the errors. There's a lot of context that an agent doesn't have about what is good and what is bad. The best way to explore what is good and what is bad is to look at the data yourself. So we recently did a study comparing all of the leading eval vendors on their capabilities with automated evals, and all of the leading vendors have some sort of feature that allows an agent to look through all of your data to find issues. We compared Arise, LangSmith, and Braintrust, and we found that they can all find issues that are obvious, But they all fail in the same ways. We use the same apartment leasing AI assistant data that we've discussed before. So all of these agents failed in the exact same way and failed to catch the same errors. For example, they all missed issues with sales objections. They missed formatting issues like having markdown in text messages. They all failed to capture problems with interruptions in their voice agent. And they also all failed to capture problems with human handoff. I'll put a link to this study in the description so you can dig deeper. But what you need to know is you need to be looking at your data so that you can refine what good looks like. It's impossible for you to know what good looks like without looking at some data. If you were just tasked with brainstorming all of this upfront, it's likely you would miss something and your evals wouldn't be really good. If you want to avoid these mistakes in practice, check out this video where I go over 8 cloud code skills that can help you improve your evals.
