Hamel HusainML Engineer Helping Companies Build AI Products

Why Your LLM Judge Should Say Pass or Fail, Not 1 to 5

9 Oct 2026 · 1 min

Watch on YouTube

0:43

In this segment, Hamel Husain argues that LLM judges should output a binary pass/fail decision rather than a Likert or continuous score. He explains that LLMs struggle with producing reliable continuous scores, and such scores are ambiguous and non‑actionable (e.g., an average of 3.2 vs 3.7 provides no clear guidance). By framing the judgment as a simple yes/no question—"Is this feature good enough to ship?"—the output becomes clear, decisive, and directly supports product decisions.

Chapters

It's very important for an LLM judge that you output a binary score. 99% of the time, you don't want to export like a Likert scale or a score of 1 to 5 or some kind of score because that introduces tremendous complexity. LLMs are not good at continuous scores, number one. Number two is the output is not gonna be clear. When you see an average score of 3.2 versus 3.7, no one really knows what the hell that means. And it's not really actionable, honestly. I've found that when you try to hide behind a score, you're not really making a decision. The frame here is, is this feature good enough to ship? Yes or no? Make a decision.