Hamel Husain explains why a 100% pass rate on AI evals is a red flag. Perfect scores usually mean you've overfit to the tests or they're too easy to matter. Build in harder evals and failures so you actually learn where your model breaks.
—And one of the things we learned is if it afterpasses frankly even like 100 tests, the odds that it will do on any random distribution of like user inputs the next 100,000 100% accurately is like very high.
When you see a 100% pass rate on evals, this is usually a really bad thing. What it means is your evals aren't telling you anything. If you're just passing every single test, you're not getting any signal on what your AI is not doing well on, and you're just patting yourself on the back. Usually when you see 100% pass rate, what it means is that you've overfit on on your evals, meaning you've kind of trained the prompt to overfit on those test cases. And that's really not good because you don't know if your AI is generalizing. It's either that or you've made your evals so easy that they're trivial to pass. Either way, this is not a good idea. What you should aim for is having some evals that fail. You want some hard evals that are even reaches that you fail on so that you have something to iterate again.
