Hamel Husain argues that 1-to-5 Likert scales for AI evaluation are unreliable because raters can't consistently distinguish between scores like 3.2 and 3.5, and aligning on what each number means is expensive and error-prone. He recommends binary pass/fail judgments tied to specific error categories instead, which force clear shipping decisions.
Stop using Likert scales for your AI evals. Likert scales are 1 to 5 scales and they simply don't work. No one knows what the difference between a 3.2 and a 3.5 is. These don't work. The reason they don't work is because they're very ambiguous. It's very hard to get humans aligned on what is a 3 versus what's a 4. The whole process of creating the rubrics can be very expensive and what ends up happening is that people kick the can down the road and they don't make a decision on whether the product is good enough. And so what ends up happening is these 1 to 5 scales end up hiding lots of uncertainty when it comes to an LLM as a judge. So don't use a Likert scale. What you want to use is a binary outcome. So you want it to either pass or fail, and that aligns with the idea that you are going to have to make a judgment call on whether this product is good enough to ship. So what you want to do is you want to scope your eval to a very specific error and you want to have it be binary pass or fail.
