Hamel Husain argues that traditional NLP metrics like ROUGE and BLEU no longer work for modern language models, and Likert scales hide uncertainty in middle values. Binary pass/fail ratings are clearer, easier to apply, and directly tell you whether to ship.
So, okay, with Rooge and BLEU, you know, these scores, they're kind of from traditional machine learning natural language processing. Okay, like it's worth thinking about what is the score? You know, like when we're talking about string similarity, for example, is string similarity like really a good metric for you to use when assessing if an answer is right or not? Probably not. It was maybe a good coarse metric to use back in the day when we were struggling as machine learning to get LLMs to produce any coherent language and producing things. But now, now we're in a different era where like, okay, we're past the— way past the point of producing like coherent English language that's like structurally syntactically correct, and the string similarity is not getting— is likely not getting you there. The Roogen Blue Score is kind of just like, it's part of the eval industrial complex of like, we just have some metrics, we're going to give you these metrics, just trust us because we're experts. You might feel like, oh, this is like some fancy stuff, but it's not like what you need. So that's like the first thing. You probably wanna stay away from these like Likert scales or like 1 to 10 score because what happens is in most cases, unless you have very sophisticated setup and you put a lot of resources into aligning these scores with humans, in 99% of cases, like it's a really bad idea to use a 1 to 10 scale or 1 to 5 scale 'cause no one knows what 4.2 versus 3.7 means, nobody. And what happens during annotation is like humans just kind of hide their uncertainty in like the middle values. And you really don't get good decision-making out of that either. Like at the end of the day, you have to ship your product. It was like, is this good or not? So you want to try as hard as you can to have binary evals, like did it pass or fail? You need to make a decision, pass or fail. People can understand that. You can action on that easier. It's much easier to align that with a human, which you always have to do with an eval, you know, cause like you see a score of 3.2, you're like, I don't know what's wrong. Like, I don't really know, but like fail, it's like, okay, failed. You know, now you can action on that. So that's That's kind of some things I wanna bring up.
