Hamel Husain argues that generic metrics like hallucination scores leave AI teams directionless. Instead, examine execution traces to find specific failures: mis-handled tool calls, query disambiguation problems, scheduling errors. Concrete errors guide improvement; abstract scores don't.
What are my favorite metrics for evals? My favorite metrics for evals are specific errors that you can observe or simulate in your application. It's not hallucination score, coherence score, anything like that. What it is, is when you look at your traces, you will observe very specific errors. You'll observe errors with certain tool calls mishandling things. You might have disambiguation issues with queries, with user queries. You might have calendar scheduling issues if your application is trying to schedule applications. Whenever I look at a company and ask them to show me their evals and they show me a dashboard, they say, okay, like we're using Deep Eval or we're using some kind of other framework. They only have generic metrics. We already know that they're in trouble. And that's really the reason why they usually call me to begin with. It's like, hey, we're feeling lost. And there's a correlation between the generic metrics and feeling lost because they cannot guide you.
