Hamel Husain argues that 1-to-5 rating scales hide problems in AI evaluation. Binary pass/fail metrics work better because they force you to define what actually matters, like whether an agent completes a task within acceptable time and steps. Likert scales mask noise instead of surfacing it.
I still try really hard to have binary evals. I try to scope it and say like, okay, like if the agent's taking too long, it failed because it took too long, or it failed because it's taking too many steps, so on and so forth. So I try really hard. I haven't gotten stuck on being able to convert like a Likert scale into binary yet. I've always been able to successfully do that over like working with all these companies. But I don't know, maybe there could be a case. If you have a good rubric that everyone is confident in, that's well tested, they're like, okay, this is a 1, this is a 2, this is 3, 4, or 5. Okay, fine, it might actually work. In a lot of cases, it just kinda kicks the can down the road and you're just hiding a lot of noise in there and it's not good for you.
