Hamel Husain explains how to trust an LLM as a judge for AI outputs. Ground trust in human-labeled data with proper train/dev/test splits to prevent overfitting. Skip raw accuracy and use metrics like true positive and true negative rates to catch when a model ignores rare errors.
LM as a judge is a really popular mechanism to use when writing your evals. But an important question is, how do you trust LM as a judge? This is part 7 of a 12-part series on AI evals. The only way to trust an LM as a judge is to measure it against a human label. It's just like anything else. You got to trust anything, you have to check it. And so what you wanna do is collect human labels about good and bad outcomes, and you want to collect a dataset. And this dataset should contain a train, test, and dev set. And the reason for splitting your dataset into these 3 chunks is to make sure you don't overfit your LM as a judge on these labels. So you don't inadvertently cheat and think that, oh, my LM as judge is so good because it aligns with my label so well. Well, you wanna make sure that it also aligns with labels that you haven't seen. And another sort of nuance here is you wanna make sure you source your few-shot examples from the train set only so that you can measure if you're overfitting or not. The next thing you need to know about how to trust your LLM as a judge is how to measure that trust. And so one metric that people use a lot is accuracy, and you do not want to use accuracy as the metric. Let me explain why. If you have an error that only occurs 5% of the time, if you just blindly guess that the error never happens, your judge would be accurate 95% of the time just by doing nothing. That's not good. That would look really good on paper, and you might fool lots of people, but it won't work for you. What you need to do is use other kinds of metrics that are sensitive to this dynamic, like true positive rate, true negative rate. This flashcard defines all the mechanics behind that, and if you want to learn more, check out the comments where I'll have more resources as well as a link to all the flashcards in this series.
