Michael Kaminsky explains why marketing science must measure incremental profit from each channel, not just correlation. Branded search looks correlated with sales but over-investing in it lowers profit. A real media-mix model predicts what happens when you change the budget and tracks whether those predictions actually pan out.
Chapters
What we care about doing in marketing science is making better marketing budgets over time. That's the goal. Can we get more profit from our marketing budget? And in order to do that, we need to understand the levers that we have as a marketing team. And if we're specifically focused on the marketing mix, that means we need to understand how much additional profit is driven from additional investment in any marketing channel, which is to say, if we spend another $10,000 on TV, how much additional profit is that going to drive. If we understand that for all of our marketing channels, we can design an optimal marketing budget. And so everyone knows that correlation is not causation. And unfortunately, in this context, it's very tricky to actually pull those things apart. It's very easy to estimate lots of different models that show lots of correlation between certain types of marketing activity and profit on the other side. But those models, because they measure correlation and not causation, they will fail to make good recommendations. Your branded search activity might be very highly correlated with sales, but if a model attributes all of those sales to branded search, it's going to recommend blowing up your branded search budget, putting in out huge bids on that. And that could lead to way overinvesting into that channel. It would be a not a profitable decision. And so really what we care about doing is being able to predict what is going to happen as we make those changes before we make them. So there's two parts of the problem here. We need to be able to predict what's going to happen and then we need to be able to predict what's going to happen as we make changes. Just predicting what's going to happen when you don't make changes changes, you only need correlation in order to do that. There are lots of great machine learning models that only learn correlations. They make very good predictions on stable systems. In the field of marketing measurement, we need to be able to make predictions as to what's going to happen when we change the system, which is to say, when we make a change to the marketing budget, what is going to happen on the other side? That's why we talk about prediction under intervention. We care about being able to predict what's going to happen as we make changes, because that's what you are doing as a marketing team. You're always making changes. And so in order to be able to make good recommendations, you have to be able to predict what's going to happen as you make changes. So that's the goal of a media mix model, in our opinion, is to be able to predict what's going to happen as we make changes. And then really a lot of the tooling that goes with that is just having a good system for being able to track what changes are made. What was the prediction? Did the prediction come true? And we build all of those tools into Recast in order to support that over time so that way businesses can see, is the model actually making these good recommendations? Do we see the results that we expect when we actually make these changes?
