In this talk, Hamel Husain addresses a common question from his evals course: whether the same large language model (LLM) can serve both as the application agent and as the evaluation judge. He explains that using the same model is generally acceptable because the judge performs a different, scoped binary‑classification task, and the key factor is how well the judge aligns with human‑labeled data. Husain emphasizes focusing on precision and recall on a held‑out test set, and only switching models if alignment fails, noting the onboarding complexity of new providers. He advises starting with the most capable model to establish strong alignment before optimizing for cost. Finally, he promotes a free, stage‑customized evals guide (bit.ly/evals‑guide) for AI product teams seeking to improve their evaluation pipelines.
Chapters
One popular question that I get in the evals course is, can you use the same model for your LLM judge as your agent? And as you can see, people want to know, can you get away with having to use a different model? One student asks, won't the same model have bias? Or should the LLM always be different from the LLM used in your application? Does the LLM that does the judging, should it always be more powerful? than the model used for your agent? One student even learned somewhere that they couldn't use the same model for the judge and the agent. And finally, people want to know if it's okay to use the same family of model. For example, if you're using OpenAI models, is it okay to use another OpenAI model or the same thing with Claude? These are all really good questions. And the question comes down to, can you use the same model for both the main task and evaluation? And for LLM as a judge selection, using the same model is usually fine. And this is because the judge is doing a different task than your main LLM pipeline. There is some research that shows that models can exhibit bias, but ultimately what matters here is how well your judge aligns with human labels. The judges we recommend building do scoped binary classification tasks. So we found that iterative alignment with human labels is usually achievable. What you should focus on is achieving good precision and recall with your judge on a held-out test set. Now, if you struggle to achieve good alignment, then maybe consider trying a different model. You should also consider the fact that onboarding a different model provider is sometimes very complicated. So if you are using OpenAI models, it can be non-trivial to now all of a sudden onboard Anthropic models. And so a lot of times this added complexity is not worth it. What ultimately matters is the alignment with human labels. As long as you achieve that, it's completely fine. You will find if there's any bias or any errors in your judge when you do that alignment with the labels. So I wouldn't worry about it until you find that it's actually a problem. Also, when selecting judge models, I would also start with the most capable model available to establish strong alignment with human labels. You can always optimize for cost later once you've established reliable evaluation criteria. Real quick, if you're building an AI product and you're stuck on how to improve, we put together this free resource on evals to help you. It draws from more than 1,000 hours of working with companies on building AI products, and it's customized to what stage you are at. So this is how you can use the guide. Go to bit.ly/evals-guide. When you open the guide, it'll look like this. And what you do is you select the stage that you're at. Maybe you're new to evals, you don't even know what evals are. Maybe you do have evals, but you don't really know what to test. Or maybe you don't trust your eval scores. Or maybe you think your product is just too hard to eval, or it's too time-consuming or expensive. Whatever the case may be, we have a detailed guide customized for what stage you're at. We've worked on it for over a year and are giving it away for free. Go to bit.ly/evals-guide and we'll send it to you.
