Michael Kaminsky argues most media mix models lack falsifiable predictions, letting consultants invent unprovable stories. He proposes testing models through forecasts: predict an outcome, run the experiment, check the result. Repeat this cycle to separate accurate models from false ones.
Chapters
Checking forecast accuracy is really important because it gives you something that is falsifiable in the MMM. Otherwise, everything in an MMM, there can just be a lot of doubt about it. The model might say that this channel is good and that could be true or it could be false. There's nowhere that we can go look up and check to see if the model is right. Oh, the model came back and said that TV's our best channel. Okay, maybe that's true, but also maybe not. And there's nothing to compare against. Maybe we've run some experiments in the past, but the experiment was a year ago and it sort of lines up, but sort of doesn't. So what do we do? And this is the problem with media mix models broadly is that, you know, historically a lot of consultants have made their money just by making stuff up, right? And they'll say like, oh, the model says this and you can't prove the consultant wrong because like, well, maybe like the model does say that, but there's nothing to compare against. There's no way to check their work. And so the problem is that if you don't have something that's falsifiable, if you don't have something that can be proved right or proved wrong, then it's very difficult to do actual science. You can of course always make stuff up and tell stories. That's like astrology effectively. We're doing astrology. We're just going to say like, oh, we're going to tell a pretty story that resonates with people and they love it and they're going to pay us money for that. But if we actually want to do real science, astronomy or physics, we need hypotheses that are really falsifiable. You could say, Yes, we can prove that this is true or not, or we can collect evidence that will help us make a decision. And so one of the great things about using forecasts in the context of a media mix model is it gives you something falsifiable. The model predicts that this will happen if we do X. Let's go do X and see if the model's right or not. And let's repeat that. Let's do that over and over again. And that's gonna help us learn if the model is plausibly true. We're gonna be able to go out into the world and generate real evidence for or against this hypothesis. There are way more false models than there are true models in the world. And so most likely without doing a ton of work, you're probably in the wrong model situation. And so we need to think about how do we break out of that? And again, that's why we think forecasting is so important is because it gives us something that is actually falsifiable, that we can actually go check in the real world and isn't just repeating our assumptions of the model back to us.
