Hamel Husain explains when to write an eval for a machine learning system. Build one only after observing a real failure, when fixing it has clear ROI, and when the error is both impactful and fixable. A medical AI prescribing the wrong drug despite having the right context is the kind of high-value mistake that deserves a dedicated eval.
Have you observed the failure with error analysis? So what does that mean? That means either it's occurring naturally in the data or you have a strong suspicion that this is something that can occur and you have done the due diligence to simulate the error. If you can't simulate the error, then this probably has low ROI. When you do error analysis or when you're looking at your data, you'll find all kinds of problems. Some of them will be very obvious to you, but if there's only little iteration, it's not really clear, well, you know, that you need an eval. But if it's like, hey, you know what, this is a very tricky problem, like, oh, the medical AI assistant is assigning the wrong drug despite, you know, having the right context. I don't really know why that happened. And also it's an impactful error, so it gives you a high value. That's where you know that this eval is going to pay off because then now you can have something to iterate against and it's also worth iterating against that error. It's important and you need something to help you make progress.
