Your LLM judge needs at least 50 human labels

23 Oct 2026 · 1 min

Watch on YouTube

1:04

In this brief conversation, Robert Ta asks for a rule‑of‑thumb for the minimum number of human annotations needed to reliably evaluate an LLM judge. Speaker B recommends aiming for at least 50 human labels, noting that people like concrete numbers to set a goal. The discussion then explains why smaller samples (e.g., three labels) are problematic: they produce high variance and noisy statistics, leading to unreliable estimates of agreement, precision, and recall, and result in large error bars. The overall message is that a modest sample size around 50 balances practicality with statistical stability for evaluating LLM judges.

Do you have like a typical amount of human labels you recommend as a minimum viable eval sort of?
It's not like a bright line rule. Try to tell people, hey, like have at least 50 labels, but that's just because people love to need a number. They will like not be able to sleep at night if I don't say like a number. That's why I say 50. At least it gives you some mental
hill to climb.
Yeah. Gives you like a goal that you can achieve. But, you know, like the less labels you have, the noisier your statistics are.
Say more about that.
I mean, like if you only have like 3 human labels, then the noise, the variance is going to be super high because your sample size is low. So when you go to compute the statistics of agreement between your human and the judge, you know, you want to measure precision and recall, not just agreement, then It's going to be super noisy. It's going to give you some estimate and whatever estimate you get is going to be, it's going to have error bars like very large, right? If you don't have a big enough sample,
because it's just 3,
this is something you can measure.