Michael Kaminsky argues MMM isn't just a report card on past spending. It's meant to forecast future revenue and guide budget allocation. The real test: whether a model predicts accurately when you actually change your marketing spend, not whether it fits old data.
I think to some extent it is true that media mix models are backwards looking, right? They're looking at historical data and finding patterns. But when you think about what we actually want to use a media mix model for, we can't limit ourselves to only thinking about backwards looking reporting. If you're running a media mix model, the reason why you're running it is so that you can improve your marketing budget allocation in the future. How can we change what we are doing to generate more revenue or more profit in the future? That's the whole point. We think a lot about using media mix models to do that for forecasting, for budget optimization in the future. You can't optimize last month's budget, right? It's already spent. We have to optimize next month's budget. Even on top of that, we also spend a lot of time thinking about how to validate media mix models and sort of the core problem with media mix models is that they're very easy to run. You can run thousands of different media mix models on your laptop in an hour. The problem is that they'll all give different results. They'll all fit the data equally well. And so the way that you think about breaking the tie The high is you think about, well, which of those models, which of those sets of results actually consistently do a good job of predicting the future? They consistently help us build better budgets for the next month, the next quarter, the next year. Even if you only wanted to use a media mix model as a report card for what happened in the past, which again, I don't think makes any sense, but let's say that you only wanted to do that, you still need some way of validating if the model is right or not. And again, media mix models, they're not magical, truth-telling machine. It's not a crystal ball. It's generally a pretty complex model, which means that there's lots of different assumptions and slightly different assumptions can often change the results a lot. And so we need some way outside of the model itself of validating if the model has actually picked up that true causal signal. And being able to consistently predict what's going to happen under intervention particularly is a good indicator that the model has picked up causal signal. If it can do that over and over again, that should build a lot of confidence that the model has picked up that true causal signal.
The Continuous Testing Loop That Separates Great Marketing Teams From Average Ones
