Hamel Husain warns that 90% agreement between an LLM judge and human evaluators is a trap. A system can hit that number by never flagging failures, hiding real problems. High agreement metrics are deceptive. Dig into what the LLM actually does, not just the number.
A lot of people go to just calculate agreement. Intuitively, it makes sense, right? Like, let me calculate the agreement between the LLM and the human. But the problem with agreement is a system is saying something is failing 10% of the time. You can agree with it 90% of the time by just saying it never fails. You'll be in 90% agreement. And 90% agreement seems really good on paper. You go into like a stakeholder meeting, it's like, yeah, I have a judge, you know, 90% agreement. You're like, okay, that sounds good. No, that sounds really bad actually. So as a product manager, if you ever see the word agreement, you need to pause and be like, hmm, let me dig into this please.
