Hamel Husain argues that data science skills are more valuable than ever for AI engineering. AI model outputs are noisy signals requiring the same statistical thinking and debugging tools data scientists already use. AI agents can write code and generate visualizations, but humans must decide which tools to apply and verify the results.
So first of all, I think that people should never get caught up on a job title. We should think about skills. What skills do we have and how valuable are those skills? What skills does a data scientist have? A data scientist has the skills of figuring out how to make sense of very noisy signals. Okay, these signals where they're not always consistent, they're not deterministic things happening in the world. Patterns that we don't even know the answer to. They're coming out of a black box of some kind. And then like, how do you know what the root cause of those signals are? How do you get insight into, you know, the impact of those? And how do you debug systems that emit signals that are very noisy? Data scientists have all the tools to be able to tackle that problem. Now, all the things I just said, this is what AI output is. AI output produces non-deterministic output text. How do you measure this text? Is it good? The thing that is producing the text is a bit of a black box. Like, we don't know. It's coming from an API, or even if it's coming from model weights, like, might as well be a black box. It's fairly complicated. How do we know, like, whether the L is doing what we want? How do we debug that? How do we sift through all this data? We just got a question like, I have this crazy agentic process, it's producing lots of tool calls, a lot of traces everywhere. How do I know what the hell is going on? Data analysis. Data analysis helps us find patterns. And so it's, uh, the statistical thinking when it comes to measuring and thinking about noise and what that means, but also like quantifying uncertainty, knowing about all the different traps and biases. But all the other tools, data tools of like cleaning data, data pipeline, all the muscle around data analysis, exploratory data analysis, like thinking about signals, all of that is super valuable in AI. I can't think of anything that's like more valuable than that. It's like AI has made data science way more valuable than ever before because now you just have like way more data and way more noisy signals that you need to reason about and debug and think about if they're working.
The ability of agents to be thought partners and co-builders when doing data science, I think, is moving us to— we've seen where agentic engineering is going now. And I think we're starting to really see the starts of what's happening with what we're calling agentic data science. And premise being that Claude code, not necessarily good at data science, nor should it be. And how do you build your harness around it so you can run parallel analyses and then have adversarial analytical path code review? and that type of stuff. And what's the role of the human in specification and verification when doing data science workflows where agents are helping you with a bunch of the stuff? You know, the silly examples being like, you know, writing and executing Pandas code or Polars or whatever it is. But the more meaningful ones, you know, having them as partners in designing analytical paths and experiments and analysis and then adversarial reviews.
So, and I don't think like just You know, the fact that AI can help you design an experiment, of course, like write your Matplotlib code, like, okay, like that saves a lot of time.
Mm-hmm.
But even if you wipe all that stuff out, there's all, you know, there's all of this judgment and skill about like what tools to apply at what time. Even if you zoom into something very specific like RAG. So the R, you know, of RAG retrieval, So retrieval is a search problem and search, this information retrieval is a very deep data science field of like, how do you make search good? How do you measure search? How do you know if, how do you like tune the search to sort of do better? And how do you measure that? There's, there's so many search metrics that, you know, and the search metric you apply depends upon your product goals. And so, and it can be very tricky to apply the right search metric depending on what your goals are. So like you have to kind of command the tools. And until we have AGI, the AI is not going to command itself like that well at a very high level. And so you'll still need a data scientist.
