In this talk, Hamel Husain addresses the common question of who should be responsible for AI evaluation within product teams. He argues that evals should not be a separate, siloed discipline but an integral part of the inner loop of AI development, with every team member paying attention to data and error analysis. The core recommendation is that a domain expert—acting as a "benevolent dictator"—should drive the evaluation process, making final calls on what constitutes good performance. Husain cautions against outsourcing evals to developers who lack domain expertise (except for coding‑focused products) and warns that having too many people labeling can make evals expensive and inefficient. To empower the domain expert, he suggests building custom annotation tools using LLMs like Codex or Claude Code that surface all necessary information in a single view. He concludes by promoting a free, stage‑based evaluation guide (bit.ly/evals-guide) that helps teams at any maturity level design and run effective evals.
Chapters
A common question I get about AI evals is who should be doing the evals? For example, one student wants to know, what should be the team structure and roles for AI development and evaluation? Another student wanted to know, how do product managers fit in? Are evals the responsibility of engineers or PMs? Or what is the role of a domain expert versus the role of an engineer? Another good question is, should evals be a dedicated role on your team? And also, how do you split work between people doing the evals and people doing the building? So these are all really good questions. So first of all, I don't think evals should be its own discipline. Eval is just fundamentally in the inner loop of how you build an AI application. Everybody needs to be looking at the data. If you make it its own separate role, then it risks becoming a silo. You definitely don't want to do that. You also do not want to be outsourcing evals to people without domain expertise. What this means is you usually should not be outsourcing evals to developers unless you are building a coding application. Developers are not the domain expert. You need the person that has the most domain expertise driving the eval, because one of the key tasks in evals is looking at data and doing error analysis. What I recommend is establishing something that looks like a benevolent dictator. And what a benevolent dictator is, is a domain expert whom everybody trusts and who can make the final call on on what good looks like. You can't always have one person in charge like this, but it's good to try really hard because you wanna eliminate as much complexity as possible. If you have too many cooks in the kitchen, too many people labeling things, it can make evals super expensive and it can make labeling super expensive. You're not trying to get to perfection, you're just trying to extract good signals from your data so that you can improve. And oftentimes you can get really far a benevolent dictator. The best way to make a benevolent dictator successful is to give them great annotation tools. These are usually tools that you create yourself with Codex or Claude Code, and they are custom-built for your data and your domain, and they render data and make it very easy to read and put all the information that an annotator needs to see in one place. So to recap, No, evals should not be a separate role. And 2, the person in charge of the evals should be a domain expert. Real quick, if you're building an AI product and you're stuck on how to improve, we put together this free resource on evals to help you. It draws from more than 1,000 hours of working with companies on building AI products, and it's customized to what stage you are at. So this is how you can use the guide. Go to bit.ly/evals-guide. When you open the guide, it'll look like this. And what you do is you select the stage that you're at. Maybe you're new to evals, you don't even know what evals are. Maybe you do have evals, but you don't really know what to test. Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval, or it's too time-consuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. We've worked on it for over a year and are giving it away for free. Go to bit.ly/evals-guide and we'll send it to you.
