Hamel Husain explains that LLM evals are systematic feedback loops that replace guesswork with concrete metrics. Without them, developers can't tell if prompt tweaks break other functionality or actually improve performance as applications scale.
What's an even deeper way of understanding what an eval is? Without evals, you might iterate on a problem and be left with guessing. Let's say you have a real estate assistant application and it's not working the way you want. It's not writing emails to customers the way you want, or it's not calling the right tools, or any number of errors. And before evals, you would be left with guessing. You would maybe fix a prompt and hope that you're not breaking anything else with that prompt. You might rely on vibe checks, which is totally fine, and vibe checks are good, and you should do vibe checks. Initially, but it can become very unmanageable very fast because as your application grows, it's really hard to rely on vibe checks. You just feel lost. And so evals help you create metrics that you can use to measure how your application is doing and kind of give you a way to improve your application with confidence, that you have a feedback signal in which to iterate against.
