Your LLM Judge Needs At Least 50 Human Labels

23 Oct 2026 · 1 min

Watch on YouTube

1:04

In this interview excerpt, Hamel Husain discusses practical guidelines for evaluating large language model (LLM) judges using human annotations. He suggests a rough minimum of about 50 human labels for a viable evaluation, noting that people often want a concrete number to aim for. He explains that smaller sample sizes, such as only a few labels, lead to high statistical noise and large variance, making metrics like precision and recall unreliable and resulting in wide error bars. Husain emphasizes measuring precision and recall rather than just raw agreement between the LLM judge and human labels, and he cautions that insufficient labeling can produce misleading evaluation results.

TranscriptAuto-generated · 1 min

Do you have like a typical amount of human labels you recommend as a minimum viable eval sort of?
It's not like a bright line rule. Try to tell people, hey, like have at least 50 labels, but that's just because people love to need a number. They will like not be able to sleep at night if I don't say like a number. That's why I say 50. At least it gives you some mental
hill to climb.
Yeah. Gives you like a goal that you can achieve. But, you know, like the less labels you have, the noisier your statistics are.
Say more about that.
I mean, like if you only have like 3 human labels, then the noise, the variance is going to be super high because your sample size is low. So when you go to compute the statistics of agreement between your human and the judge, you know, you want to measure precision and recall, not just agreement, then It's gonna be super noisy. It's gonna give you some estimate and whatever estimate you get is gonna be, it's gonna have error bars like very large, right? If you don't have a big enough sample.
Because it's just 3.
This is something you can measure.