Michael Kaminsky and a colleague explain why MMM models overestimate advertising's impact during demand spikes. Marketers spend more on ads when demand is already high, creating bias that makes the model credit ads for sales that would have happened anyway. They use simulations to test how well MMM catches this problem.
Chapters
So I know that some of the hardest problems is dealing with multicollinearity, for example, or the endogeneity of the spending where you might spend more specifically in cases where there is more demand. So sometimes I will simulate data that has like a baseline demand, a seasonality, and then I'll have it where spend is tracking alongside that seasonality. And this is something that you can imagine marketers are doing. If you're working for an e-commerce, there's a ton of buyers out there during the holiday season. So you ramp up the ad spending, you ramp up the ads.
Talk to any marketer and they will tell you they do this. It's not like, it's not a secret. They will tell you that this is part of the plan. Exactly.
And it makes sense, right? You know, this is when the buyers are out there. But now imagine this, when, if you're looking at the data, if you were just naively looking at this data, you see higher advertising spend and you also see higher sales. But of course, you have this omitted variable bias that there's increased demand out there in the market because it's the holiday season. And so I would start simulating these things in the code that there is endogeneity or that what happens when we add a second media channel, but now these two media channels are correlated. So we have multicollinearity in the data. How well do the models actually pick these things up?
Amazing. I love this approach. And this is something that we do a lot at Recast. We call this process a parameter recovery exercise, which is to say You simulate data with some known parameters and then you push it through a model and you try to recover the parameters on the other side and you check how well the model does. And this is like what you're describing is what we recommend to anyone evaluating any type of media mix model. Because as you note, the core problem with media mix modeling is that the truth that we care about, the true incrementality or the true causality of all of our different marketing channels is unknown and unknowable. We cannot, there's nowhere that we can go look up what the answer is and say like, yes, the model got it right or it didn't, right? With real data, with the real data that we're using for our business, we just can't know that. And so parameter recovery helps us understand where a model works well and where it doesn't under ideally different simulations. And so what I tell people is when you're evaluating one of these models, you should always try to do it in such a way so that it breaks. And if you can't find it, you've screwed up, right? Because every model has limitations. Every model will break down under some set of assumptions. And so your job when you're actually evaluating these things needs to be, how can I simulate data in such a way that it breaks the model in a really meaningful way? And then that lets you start to reason about, well, is that important or not? Right? Does that matter or not?
