Hamel Husain explains how to sample trace data when evaluating AI systems. Start with random sampling and clustering to find unknown failures, then use metadata like latency and existing evals as filters. Direct user feedback like thumbs up/down gives the clearest signal of what's broken.
Looking at data is the most important thing you can do if you're building an AI product. And what we mean by data is looking at your traces. But what traces do you look at? This is part 8 of a 12-part series on AI evals. And so today we're going to talk about how you sample these traces. So here are the different strategies that ranges from exploring traces all the way to using signals that you already have that certain traces have errors. And you should leverage all of these, and these are a spectrum. So the first thing that you should always consider on the explore side is random sampling. This is the best way for you to encounter unknown unknowns, things that you don't anticipate or wouldn't have thought to look for. The second thing that is also exploratory is clustering. You can use a clustering algorithm either on embeddings or other features of your traces and group them together to see if you find any interesting patterns. Another exploratory technique but is getting into signals now is to use metadata and data analysis. So look at statistics such as latency, number of turns, so on and so forth to find outliers and see if those correlate with bad behaviors or errors. Next, we can get into signals. So if you've already written evals and your evals are good, you can use those evals as classifiers. You can apply those evals to your traces to see if it triggers any of those evals. And you'll have a filtering mechanism to see which traces might be good to look at. And then lastly, you should never ignore feedback. If you have explicit user feedback, thumbs up, thumbs down, other kinds of feedback, you should always pay attention to that. If you want to learn more about evals, including this subject of how to sample traces, I've included a link in the comments along with a link to download all of these flashcards.
