Hamel Husain shows how to use an LLM as an automated judge for subjective work like sales emails. Collect human-labeled examples, fine-tune the judge to match those judgments, test it against held-out data to verify it works, then deploy it to score new emails at scale. Once validated, the judge becomes a reliable quality check.
If you're building an AI sales assistant that writes outbound emails, you might want those emails to be persuasive. But how would you grade those emails given that persuasiveness is subjective? One technique is for grading these subjective outputs is to use an LLM as a judge. An LLM as a judge is simply another LLM that grades your LLM. And why is that okay? Well, first you have to collect a human-labeled dataset and you have to make your judge align with that human-labeled dataset and you have to grade the grader. Once you do that, you You can now trust the LM as a judge, and now you have a grader that you can apply against all of your conversations, and you can iterate on to make your system behave the way you want.
