0:37
Hamel Husain explains why automated evaluations matter for AI apps. Without them, you're guessing whether prompt changes work or break things. As your application grows, informal testing becomes impossible to manage.
Hamel Husain explains why automated evaluations matter for AI apps. Without them, you're guessing whether prompt changes work or break things. As your application grows, informal testing becomes impossible to manage.
Without evals, you might iterate on a problem and be left with guessing. And so let's say you have a real estate assistant application for— and it's not working the way you want, it's not writing emails to customers the way you want, calling the right tools, or any number of errors. And before evals, you would be left with guessing. You would maybe fix a prompt and hope that you're not breaking anything else with that prompt. And you might rely on vibe checks, which is totally fine, and vibe checks are good, and you should do vibe checks. Initially, but it can become very unmanageable very fast as your application grows. It's really hard to rely on vibe checks. You just feel lost.
