Hamel Husain argues that error analysis, a disciplined review of about 100 model outputs, is the highest ROI activity for AI engineers. Sampling and journaling observations without diagnosing root causes reveals hidden failure modes and clarifies product requirements in under 30 minutes.
The lowest hanging fruit when it comes to evals is to start doing error analysis. You can make progress in less than 30 minutes and get tons of value. It's probably the highest value activity that you can engage in. And error analysis is just a fancy way of saying, look at your data, but look at your data in a structured way. Don't just kind of randomly look at it. What you want to do is sample some of your traces and start writing notes about what is going wrong. It's kind of like open journaling. When you write those notes, you learn quite a lot. And your goal is not to diagnose a problem or do root cause analysis. You just want to observe. When you do that for, let's say, 100 or so traces, you will learn quite a bit about how your application behaves, but then also you'll refine and be able to externalize what you want. The research shows that people aren't good at defining their requirements with AI products until they react to what the AI is doing. And so error analysis is secret superpower of any AI engineer or product manager who is building an AI product. It's the fastest way you could start making progress.
