Hamel Husain breaks down three types of automated evaluations for AI products: code-based assertions for deterministic checks like date verification, LLM-as-a-judge for subjective text outputs, and guardrails that block unsafe responses before users see them. Each serves a different purpose and requires different validation approaches.
There's 3 types of automated evals that you should know about if you're building AI products. So the first kind of eval is really simple. It's a code-based assertion. This is very similar to what you might have in software engineering where you have a ground truth of some kind that's easy to assert against. You can do a deterministic check such as, is a date correct, or is a field that you're trying to extract have a correct value, etc. The second type of automated eval is an LLM as a judge, and this is the most common kind of eval when it comes to AI products. This helps you evaluate subjective outputs like text in writing that is not easy to write assertions against. What you need to know about LLM as a judge is that it's very important that you align the judge to human labels and you measure the agreement between human labels and the judge so that you can trust it. The third kind of automated eval is a guardrail. A guardrail is a special kind of eval that sits in the request-response path to the user and blocks the user from seeing unwanted output. And usually these evals are fast and very reliable. So these are the 3 kinds of evals you need to know about. If you want to know more about evals, I'll put more resources in the comments, including the entire set of these flashcards so you can learn more.
