Michael Kaminsky explains why a model that fits your data perfectly can still give you completely wrong causal answers. Most data scientists are trained for prediction, not causation, and mistake good fit for causal validity. John, an economist, says the fix is learning experimental design and how to build valid control groups when you can't randomize.
Chapters
you're talking about this exercise where the model with the higher in-sample fit metrics, it fits really well to the data in-sample, but it actually doesn't get the causal inference correct. This is critically important. It's really important cuz we're, what we're doing is causal inference, not just model fitting. And there are contexts in which looking at goodness of fit for models makes sense and is a fair way to evaluate those models. but in the context of causal inference, it almost never is. And I think a lot of what's going on is that a lot of data scientists, especially recently, have come up through this world of like machine learning and prediction, right? Whereas that is so different from the world of causal inference and econometrics. I'm wondering if you have other things that data scientists should be looking out for as they're making that transition into doing causal inference or working with mediabixed models. When maybe they don't have a formal econometrics training.
I have a lot of followers who are coming out of data science bootcamps, who are coming out of a master's or an undergrad in data science, and they are telling me, John, you're sitting here talking about causal inference problems. We are not learning this in school. Like, we are not learning this in our programs. I guess it makes sense because these programs are giving people what they want. I, I actually feel in a lot of ways that I got very very lucky. I went through an econ PhD, but my focus was on public policy. And these things fit so well because the government a lot of times cannot randomize. We're not going to randomize what student loan interest rates are, right? We're not going to randomize what the minimum wage is going to be. And so by nature, a lot of the public policy things that we study necessarily need this observational causal inference. And so we just have been trained to think Wow, we wish we could run an experiment, but we can't. What are ways that we can get a valid control group out of this? And so when people are trying to get into this, I think the best thing they can do is first get a strong foundation in experimentation. Do you understand why we need experimentation? Can you do experiment design? For me, step one is understanding deeply A/B testing. Analyzing and designing experiments. And from there, you can start to build into some of this causal inference.
