Michael Kaminsky argues most marketing teams use MMMs without validating them through randomized trials, relying instead on regression that offers no accuracy guarantees. He says these models are often simply wrong, not just potentially wrong, and their outputs shouldn't be trusted without proper testing.
I think you and I broadly agree that there's a lot of deep problems marketing mixed models or observational causal inference when it comes to measurement and especially in the field of marketing. You've commented that marketing teams are going to keep using MMMs without validating them and that this is sort of a big, a big problem. What's going on here? Why is this a problem? Why do people keep using MMMs without validating them? Like what's the background that's causing us to see this over and over over again.
Fundamentally, I think that not validating is probably too strong. Not validating via randomized trials, absolutely. The vast majority of usage that I see in the wild, people don't, you know, run randomized trials. Recast has launched a Geolift product, but everyone sort of looks at those and says, do I really need to do that? That's an interesting question, right? A perfect world where I had infinite resources, Absolutely, I should be validating everything. In a world where I have to make resource trade-offs, does that meet the bar? The articles I've written on validation, I've talked a lot about how your model is potentially wrong.
And I would go even further. You said potentially wrong. I think that understates the case. I think probably wrong is like the right way to think about this problem. You get no guarantees just from running data through a regression.
I don't think that people are actually just taking the model and using it. There's a hierarchy of tests that ultimately wind up at RCTs. But first and foremost, I build my first MMM and then I look at the output. And if it tells me something that all of my smart humans say is stupid, I am extremely skeptical, right? So there is that first-pass filter of I have experts, they know my business, they've been doing my advertising strategy. There's a second pass of validation, which is you change your spend and you notice a change. You do know that you have increased your spend on Meta or you have decreased your spend on Snapchat or something like that, and you do see an output. There's a third pass of validation, which is at least in the field I'm in, which is digital entertainment. Now I have an MMP in the mix, right? Their job is essentially to tell you more about where your users came from. And they're not doing it in the MMM sense. And so you wind up saying, okay, I've got an MMM. Question 1, did I do the MMM right? Question 2, does the output of the MMM meet these various sniff tests? Does it make sense with respect to what my smart humans think? Does it make sense with respect to the known variations in spend that I have undergone over the past 6 months? Does it make sense with respect to my mobile measurement partners and the last-click attribution?
