Hamel Husain argues that organizations should appoint one qualified person to evaluate AI outputs instead of requiring committee consensus on every decision. A large review committee makes the process too expensive and slow. One trained evaluator can gather enough context to judge AI behavior effectively.
This is why I tell my clients to appoint one benevolent dictator to doing annotations for eval. And it might sound like a funny word, benevolent dictator. This is because evaluating AI can get very expensive. You wanna keep it tractable and be pragmatic. And the last thing you want is an entire committee to be looking at every trace. That'll make the process too expensive. And what you want to try to do is you wanna try to delegate to the right person that can make the call on whether an AI is doing the right thing or not. And usually it's the case that you can find a person like that, or you can train a person like that who can gather enough context to do that task. And for the vast majority of people, that's what we recommend. For more information on this, look in the comments.
