Hamel HusainML Engineer Helping Companies Build AI Products

Most people try to write their evaluation criteria upfront. It doesn't work.

24 Mar 2026 · 0 min

Watch on YouTube

0:22

Hamel Husain argues that writing evaluation criteria for language models upfront doesn't work. People only clarify what they actually want after seeing the model's behavior and reacting to its outputs, then refining their criteria through detailed notes and critiques.

There's a paper called Who Validates the Validators? And the research shows that people are really bad at writing specifications or requirements until they need to react to what an LLM is doing to clarify and help them externalize what they want. And it's like only going through this process of sort of writing detailed notes and critiquing things that you can then like start refining the LLM judge.