Hamel HusainML Engineer Helping Companies Build AI Products

The Two Biggest Mistakes Teams Make With Evals

15 Oct 2026 · 1 min

Watch on YouTube

0:41

In this brief video segment, Hamel Husain highlights two prevalent errors that teams commit when conducting evaluations (evals). First, they assume that purchasing an off‑the‑shelf AI tool will automatically handle all evaluation needs, overlooking the fact that such tools rarely work out‑of‑the‑box and require careful integration and oversight. Second, they skip the crucial step of examining raw data and execution traces, which, according to Husain’s consulting experience, consistently reveals hidden problems and informs effective fixes. He stresses that a disciplined, data‑driven approach is essential for reliable eval outcomes.

Chapters

The top one is, hey, I can just buy a tool, plug it in, and it'll do the eval for you. Why do I have to worry about this? We live in the age of AI. Can't the AI just eval it? That's the most common misconception. And people want that so much that people do sell it, but it doesn't work. The second one that, you know, I see a lot is, hey, just not looking at the data. In my consulting, people come to me with problems all the time. And the first thing I'll say is, let's go look at your traces. And we always, 100% of the time, learn a lot and figure out what the problem is. I think people just don't know how powerful looking at the data is.