Hamel HusainML Engineer Helping Companies Build AI Products

The Counterintuitive Truth: Good Evals Should Show Failures

26 Nov 2025 · 0 min

Watch on YouTube

0:25

In this segment, Hamel Husain argues that an AI evaluation suite that yields a 100% pass rate is itself a failure. Effective evaluations must surface failure cases, providing concrete signals about what aspects of the model are still broken. These signals guide iterative development, turning green checkmarks into actionable goals rather than mere self‑congratulation.

If nothing has failed when you run your AI evals, meaning you have 100% pass rate, that means your AI evals have failed you. Your AI evals should be giving you some signal into what is wrong, and you should always strive to have some failure cases that you can iterate against. The point of it is not just to have a bunch of green checkmarks everywhere and pat yourself on the back. The point is to understand what is currently not working and give you a goal that you can drive towards.