Hamel Husain warns against using one LLM to judge another without first verifying the judge itself. Skipping this meta-evaluation step creates unreliable metrics that erode stakeholder trust when reality doesn't match the dashboard numbers.
The main problem that people encounter when doing LLM as a judge is they just prompt another LLM to kind of judge what your LLM did. And they say, is it good? Now that should, that should be suspicious, right? Like, why is that okay? Why are you just gonna tell another LLM to tell me if it's okay? Mm, I don't know. And you would be right to be very suspicious of that. And there is an answer to that, and the answer is you can measure the judge. It's a meta evaluation problem. It's like you need to measure if the judge is good. It's very important. You don't want to skip that step because if you have a bunch of judges, LM judges floating around and your stakeholders on the dashboard, your stakeholders looking at them and everyone's like, no one's gonna understand what you're using LM judge anyways. They're just gonna look at your metric. And when there's enough gap between reality and your metrics, no one's gonna trust you anymore. Cuz if people don't trust your evals, they won't even trust you.
