In this talk, Hamel Husain addresses the common challenge of persuading engineers, leadership, and product teams to allocate time and resources to AI evaluation (evals). He argues against hard‑selling the concept and instead recommends leading by example through systematic error analysis. The process involves reviewing 50–100 model traces, documenting top failure modes, quantifying their frequency, and crafting narratives about surprising user behaviors. By regularly sharing these findings, the fixes applied, and the resulting drop in error rates, a practitioner becomes the go‑to expert on the product’s AI performance, thereby generating organic interest and buy‑in. Husain emphasizes that dashboards alone are insufficient; the story behind the numbers—how errors frustrate users and impede business goals—must be communicated. He also notes that upfront ROI is typically speculative, akin to other ML projects, and value must be demonstrated over time via experiments. Finally, he promotes a free, stage‑customized eval guide (bit.ly/evals-guide) that helps teams at various maturity levels adopt eval practices.
Chapters
A common question I get about evals is how do you convince your team to invest in evals? One student asks, how do I convince the lead developer and head of AI that this process is worth investing in? Another student asks, how do you make the case to carve out time to do error analysis or read data? Or how do you convince leadership of the value of evals in the first place? And how do you get buy-in? Are there tactics that you can use to demonstrate the value of the work of evals? One student even comments, it's been an uphill battle convincing engineers to invest in evals. And how can you make a business case and quantify the ROI upfront before you ask people to invest in it? So these are all good questions. So my advice is this, is don't try to hard sell your team on evals. Instead, do error analysis. So look at 50 to 100 traces and take notes about what issues you find. Error analysis is a structured way of looking at data effectively that allows you to find issues in your AI product. And so what you should do after error analysis is you should list the top failure modes you discover. You should quantify how often those high-impact errors are happening and tell a narrative about the surprising ways that users are interacting with your product, because you will find that users are doing surprising things with your product. If you take the time to read traces. And then as you fix those issues, you should update your team on progress on how you fix the issues you found. And you should do this on a regular cadence. You should do data analysis, present your team with issues you found, and then how you fix them. By doing this, you demonstrate lots of value and it generates lots of interest in your process because quickly you become the most knowledgeable person on your team about how your AI product works. And what the issues are. And this is the best way to sell your team on evals, is by leading by example and starting with looking at data. Doing this is the way I convince organizations to invest in evals, is to do this regular data analysis and consistently show value by raising the issues, but then also showing how you fix them. The important thing here is don't just show dashboards with metrics. Explain to your team what you're finding in your data. And the narrative on what users are doing, how it's frustrating business progress, and how you can fix it. That shows a nuanced understanding, and it shows that the process creates value. And then finally, when you fix an issue, you want to show how that error rate went down and describe what you did to fix it. If you do that over and over again, people will get very interested in what your process is. So essentially, you want to let the results lead the conversation on evals. You don't want to try to come with an academic argument about why evals are good or why you need them. And to answer the question about, hey, can you quantify ROI upfront? You usually cannot. It's very similar to classic machine learning and data science projects where the value is speculative and you need to conduct lots of experiments. And this looking at data and conducting experiments, you need to prove its worth over time before you convince people and rewire their brain. Hey, this evals process is actually good. So the best way to convince people to invest in evals is to lead by example and look at the data. Real quick, if you're building an AI product and you're stuck on how to improve, we put together this free resource on evals to help you. It draws from more than 1,000 hours of working with companies on building AI products, and it's customized To what stage you are at. So this is how you can use the guide. Go to bit.ly/evals-guide. When you open the guide, it'll look like this. And what you do is you select the stage that you're at. Maybe you're new to evals, you don't even know what evals are. Maybe you do have evals, but you don't really know what to test. Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval, or it's too time-consuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. We've worked on it for over a year and are giving it away for free. Go to bit.ly/evals-guide and we'll send it to you.
