Hamel Husain argues that AI teams waste time debating what "good" means. Instead, review your model's actual outputs, document what fails, and build your quality criteria from those concrete examples. This stops the goalposts from moving as people get used to the system.
If you're building an AI product, it's really common to get stuck in arguing what good looks like. You should use this workflow instead, which is called error analysis. Research shows that there's a phenomenon called criteria drift, where when you look at data and outputs of your AI system, your criteria of what you want shifts significantly in reaction to what the LLM is able to do. And so what you want to do is start looking at data and start annotating data. In the process of you writing notes about what is working and what is not working, refines your rubric and it helps you what's actually wrong, and it also clarifies what is good and what is bad.
