Hamel Husain argues that offline machine learning tests miss what actually happens in production. A/B experiments and real user behavior reveal where conversations break down. Data scientists need to combine test results, live metrics, and narrative insight to explain why users do what they do.
evals are only offline tests. You need online tests. You need to measure what's happening in production, how people are reacting to your product. And those metrics can give you some— another lens. Why are conversations dropping off at this point in the user journey or whatever? And you can try to debug that and say, I need some offline tests. I didn't know this was happening or whatever. So you need like a whole suite of things. And the limit, it devolves into being a data scientist because this is exactly what a data scientist is even before LLMs, a data scientist is someone that helps you understand your product from a data, from a data perspective, using all of these tools at large, doing product analytics, data analytics, offline tests, online tests, and figuring out where the story is, why things are going, you know, why things are happening.
