Hamel Husain argues that generic metrics like ROUGE and BLEU don't predict real product failures. Build evals from error analysis instead: measure specific failures like calendar scheduling or mis-sent emails, use binary outcomes, and make sure your judge is verifiable.
The biggest mistake that you can possibly make when writing evals is to use generic metrics. So don't use generic metrics. Examples of generic metrics are ROOG, BLUE, faithfulness, helpfulness, tone, you name it. Vendors will offer these generic metrics on a platter to you, and usually this is a bad idea because they don't tend to correlate with actual failures that are in your product. You want application-specific metrics. Such as calendar scheduling failure or a widget rendering issue, or you email the wrong person. Whatever it is, you want application-specific metrics. And the way that you find these application-specific metrics is error analysis. So what makes a good eval metric? Here's a checklist. Number 1, measure an error you've observed through error analysis. Number 2, the eval should measure a non-trivial issue that you're going to iterate on. That way the eval is going to give you high ROI. Number 3, the metric should be scoped to a specific failure. It shouldn't be a catch-all for any kind of error you can find. It should be for one specific failure mode. The next one is instead of using a 1 to 5 scale or a 1 to 100 scale, make your evals have binary outcomes. This is easier to measure and easier to align and easier to action on. And lastly, you want these evals to be verifiable. If you're gonna use an LLM as a judge, you need to verify that judge. There's a lot of detail that we can't cover in this card directly. If you wanna learn more, check out the comments and the links where I'll give you more resources and also a set of all of these flashcards that you can go through. If you're building an AI product, think about the last time you changed your prompt. Did you have a good way of measuring if what you changed improved things? If not, evals are the highest ROI skill you can learn. Below, I'll share a link to our upcoming cohort where we will teach you how to implement eval workflows used by the top AI teams, as well as give you lifetime access to office hours so you can get unstuck.
