Hamel Husain argues LLMs should give binary accept/reject decisions instead of rating scales, which add unnecessary latency and complexity. A single domain expert then interprets the result, bringing the contextual knowledge that makes the judgment actually useful.
When you go to building an LLM as a judge, you need a binary score. You don't want to think about, is this like a 1, 2, 3, 4, 5, like assign a score to it. You can't, that's going to slow it down. So don't do that.
Just to make sure this benevolent dictator point is really clear. Basically, this is the person that does this note-taking and ideally they're the expert on the stuff. So if it's law stuff, maybe there's like a legal person that owns this. It could be a product manager. Give us advice on who this person should be.
Yeah, it should be the person with domain expertise. So in this case, it would be the person who understands the business of leasing, apartment leasing, and has context to understand if this makes sense. It's always a domain expert, like you said. Okay, for legal would be a law person, for mental health, it would be the mental health expert, whether that's psychiatrist or oftentimes it is the product manager.
