Hamel Husain breaks down three categories of LLM failure: comprehension, specification, and generalization. Comprehension means the model actually understands the task and its context. Most practitioners skip the systematic data analysis needed to find failure patterns, often because they lack formal training, but this foundational work is essential.
When thinking about LLM evals, there's 3 buckets of general failures, and these 3 buckets are comprehension, specification, and generalization. Right now, let's talk a little bit about comprehension. That's just, do you understand what is going on in your application? Did you do data analysis and specifically error analysis? And you understand what patterns of failure are before you jump into writing tests or doing anything else. This is the one place that people skip because this is not something that people have been given training in unless you're a data scientist or machine learning engineer, but it is the foundation of a lot of activities if you're going to try to make your AI better.
