Hamel Husain argues error analysis is the highest ROI work for AI products. Collect execution traces, annotate failures by hand, group them into failure modes, then prioritize by frequency. Start with 100 traces and keep going until patterns stop emerging.
In my experience, the highest ROI activity that you can do when building an AI product is error analysis. So what is error analysis? Really consists of 4 steps. So step 1 is you need to collect traces. You can do that with an observability tool of your choice. It really doesn't matter, but you need to collect all of the logs of your AI application somewhere, like in a database. The second step is annotating traces. In the literature, this is called open coding. Really doesn't matter what you call it, What you need to do is go through different traces one at a time and write notes about what you see that's wrong. And this is just open journaling about things that you might be seeing that's wrong. Third step is group and categorize. So once you've written a bunch of notes on traces, what you want to do is cluster those notes and categorize them into sort of failure modes. So you can use an LLM to do this, just keep it simple. And the fourth step is to prioritize the error. So after you've grouped everything, you can count them and you can see what errors happen the most and you can use that to guide your decision making. On what to work on. So it's really this 4-step process, and the question always comes up, how many traces should you look at? We recommend starting with at least 100 traces to make sure that you're looking at enough, but you want to keep looking at traces until you feel like you're not learning anything new. This is the beginning of a series of flashcards on AI evals. If you want all the flashcards, the link is in the comments.
