Hamel Husain argues that 1-5 scales fail for LLM judges because people default to middle values and can't agree on what numbers mean. Use binary checks instead: narrow, specific questions like "Is the response polite?" that give clear pass-fail signals for shipping decisions.
If you're using LM as a judge, one thing you should stay away from is using Likert scales or scales from 1 to 5. This is part 6 of the eval flashcard series. So binary scales are preferable to Likert scales, and here's why. Likert scales tend to hide failure to make crisp decisions because if you're unsure about whether something is good or not, people tend to gravitate towards the middle values, and that's not a good thing because you have to make a decision on whether or not you're going to ship a feature or not. Also, having Likert scales tends encourage scope creep of your evals and make them too broad. Most importantly, it's really hard to align Likert scales with annotators and get everybody on the same page of what is a 1, what is a 2, 3, or 4. For that reason, it's a really bad idea. What you want to do is break down your evaluators into scoped problems that can be binary failure modes. So instead of a broad LLM as a judge that produces a Likert scale value like 1 to 5, What you want is some specific error, like, is it polite? Is the scheduling happening correctly? Is the human handoff handling correctly? And you'll find that these are not only easier to align, but they're more actionable because you'll immediately understand what it means and what to do about it. If you like learning about evals, I put additional resources in the comments, including this entire set of flashcards that you can download. If you're building an AI product, think about the last time you changed your prompt. Did you have a good way of measuring if what you changed improve things? If not, evals are the highest ROI skill you can learn. Below, I'll share a link to our upcoming cohort where we will teach you how to implement eval workflows used by the top AI teams, as well as give you lifetime access to office hours so you can get unstuck.
