Hamel Husain explains why no LLM can judge perfectly: build a ground-truth dataset of about 100 labeled examples to measure the model's false positives and false negatives, then decide what error rate your business can live with. Some noise is always there.
Okay, so one thing to keep in mind is like with your soft checks, which is like LLM as a judge, is you want to— you can scientifically measure how good the judge is by comparing it to labels. And what I mean by that is like, okay, you should assemble a dataset of human labels that is like ground truth that you know is like, is this thing good or bad? And then you can see, you can measure like how much noise does the judge have. And if the dataset doesn't have to be, you know, if you have like 100 or so labels, for example, you can measure like the noise in the judge. There's different kinds of noise, but it'll always have noise. And you just have to be okay with the noise. You can try to tune the judge to have less noise, but there'll always be some noise and you have to tune it to where you are, you have to make a business decision like, okay, is this noise tolerable for me? There's really no such thing as like a judge that will just be 100% like blindly trusted, like with no noise at all. It is like a classifier. It's like a black box classifier. What I mean by classifier is like, just think of something that tells you if something's a stop sign or not. Okay, like even if you can get that to be very reliable, there'll always be an edge case somewhere where it's going to give you like false positive, false negative. So you have to be okay with that. You have to bake that into your decision-making of like, okay, this eval is, you know, has this kind of characteristic and you know, that's the best you can do with an LLM judge.
