Hamel Husain explains how the evals course now uses agents to help review data, a task people usually skip because it's tedious. The team worried students would stop thinking critically, but settled on a hybrid approach where AI assists and humans stay in control, matching how agents work in practice.
I've taken your course several times now, very excited to participate in the upcoming cohort. And I am wondering how you've had to change it and wanted to change it with everything that's been happening with agents now.
One of the most important changes is how do you do evals with agents? So one of the biggest bottlenecks in doing evals is looking at data, and people don't want to do it. So what we've tried to do is think really deeply about evals and where it makes sense for AI to help you. And at first we resisted it because we were like, well, everyone's going to just turn off their brains. They're not going to look at any data at all. They're just going to tell Claude just to like do everything and it's going to go off the rails. But then we kind of said, okay, you know, we can't really do that. In reality, you're going to be using a hybrid approach where the AI helps you just like you do with anything else in life.
