Hamel Husain warns against three eval mistakes: using generic automated metrics instead of testing your own failure modes, trusting an LLM judge without calibrating it against human labels, and chasing 100% pass rates that stop giving you useful signal for improvement.
Don't make these AI eval mistakes. The first mistake is choosing an automated eval metric or framework that just completely automates it away from you. And it's really tempting. You have two buttons that you can press. You can go down the path of looking at your data, finding what errors are specific to your application, or you can press that shiny auto eval button. It's very tempting. Don't press that because what you'll get is a bunch of generic metrics that don't correlate to anything that's wrong with your application. That's number one. Number two is if using an LLM as a judge, what you need to do is you need to measure that judge against human labels. You can't just take the judge at its word because what you'll get is just noise. What you'll often get is, you know, people won't be able to trust that judge. You won't be able to trust that judge. That's going to lead to a very bad outcome. The third one is aiming for a 100% pass rate on your evals. So aiming for a 100% pass rate on evals, getting that green checkmark is really tempting as a software engineer. You want to see those green checkmarks. You don't wanna do that for evals. If you see 100% pass rate, that means you've saturated your evals. They're not telling you anything. What you want is you always want evals that you're failing at to let you know how to get better and give you some signal while you iterate. I've created a series of these flashcards about evals. If you like these flashcards and you wanna download them all, check out the comment in the links.
