Hamel Husain explains why LLM judges often fail to match human judgment, then walks through a three-step fix: collect human-labeled examples, split them into train/dev/test sets, and use each to build and validate a judge that actually correlates with how humans score.
Most LLM judges that I see in production have a massive blind spot. And the reason they have this blind spot is because they are not aligning the LLM as a judge with human labels. It's just a prompt and a prayer, and you don't want to do that. What you want to do is align your judge with human labels. What you first want to do is gather your dataset, is a bunch of traces that have human labels, and you want to split that dataset into 3 parts: train, dev, and test. Your training set is what you will use for getting the few-shot examples for your prompt. The dev set is what you'll use to hill climb your evals or the alignment with the human labels. And your test set is what you'll use to make sure you haven't overfit. There's a lot of details there. I'll share a blog post that shows you how to do it step by step.
