Michael Kaminski explains how to validate marketing mix models through backtesting and holdout accuracy tests. These methods catch unreliable models before they guide real marketing decisions, ensuring your recommendations actually work.
How can we avoid overfitting in our marketing mix model? How can we build confidence in the recommendations that we're making to the marketing team? The answer Backtesting welcome to Modern analytics for marketers, our YouTube series that helps marketers learn and put into practice modern marketing measurement frameworks. This is our third episode in our series on marketing mix modeling and in particular how to validate MMMs. Today we are going to talk about one of the most important methods for validating holdout forecast accuracy and backtesting. I'm Michael Kaminski, an econometrician, entrepreneur and marketing science researcher. Welcome to Modern analytics for Marketers, presented by Recast. First, what do we mean by hold down accuracy and backtesting? Well, the idea is simple. We want to evaluate how well our MM model can predict the future. To do this, we train the model using data up to some time period in the past, like three months ago, and then we ask the model to predict the next three months. The model hasn't seen those three months of data, but we have so we can evaluate the model's forecast accuracy. It's called holdout forecast accuracy. Since we have held out the last three months of data, this is the helpful to us since if we do it correctly, this forecasting exercise can tell us how well we might be able to forecast the next three months into the future. And those are truly in the future. So we want to know how well we can do that. Another application of this idea outside of marketing is in stock trading. When a hedge fund is evaluating a new stock trading algorithm, how do they evaluate if it will work or not? How do they know if it will actually make them money? Hedge funds have sophisticated systems for doing backtesting, which is to say these are the environments where they can test their algorithms or trading strategies on historical data to see how they would have done in the past so that they can evaluate if the algorithm or strategy would have made them money. If the backtesting results are positive, that gives the fund conviction that they can go and deploy this new algorithm in the real world to generate real profits. As we discussed last episode, MMM is a causal inference problem where the true relationships we care about measuring are unknown and unknowable. What we want to validate is if the causal relationships estimated by the model actually hold in the real world, holdout forecasting accuracy can help us do this. So when can holdout forecast accuracy actually help us validate if the model has truly estimated the underlying causal relationships? At Recast, we like to say if the model can consistently forecast the future. In a system without information leakage in the face of exogenous budget changes, then the model likely has picked up the true causal relationships. That's a bit of a complicated sentence, so let's break down what that actually means consistently. Means we can do it over and over and over again over different time periods. We aren't just getting lucky with a single snapshot in time. Without information leakage means that we've set up the forecasting challenge correctly. We don't have access to any information that gives away what we're trying to forecast or predict. In the face of exogenous budget changes means that we are making predictions while manipulating the system itself. If the budget isn't changing, then we could make good forecasts by getting a read on correlation instead of causation. We need to prove that we can make good forecasts when things are changing in order to know that we've actually gotten the model right. Let's talk about what a good holdout forecast accuracy system looks like. The system should be Continuous holdout forecast accuracy isn't a one and done check, but rather should be applied at multiple time points during the initial model build and then in an ongoing way as the model is refreshed at the initial model build time. We at recast recommend backtesting the MMM model as of 30, 60 and 90 days ago as a bare minimum. And then from that initial model build forward. You want to make sure that your model's predictions continue to perform well into the future so that as the marketing team acts on your model's recommendations, they are going to do that right. You can see how well the predictions hold up. This should ideally be built into some fully automated system so that it's easy for you and your stakeholders to to monitor this performance over time. The ideal system should take great pains to avoid information leakage. Information leakage means that the model is able to cheat by using information in a forecast that wouldn't actually be available at model time or that tells you too much about the dependent variable. This is a really thorny problem in marketing mixed models because of how dynamic and interrelated the components of the system are. One good example of information leakage in the context of MMMs is happens when a model includes a variable for something like website traffic or sessions as an independent variable in the model. The problem with including website traffic in the model is that that variable isn't truly independent of the dependent variable we're trying to model. If the conversion rate is constant and you tell me how many sessions you have? Well, it's trivial for me to do a calculation of exactly how many conversions you got. This means that if you include website traffic in the holdout forecasting exercise, you're allowing the model to cheat since there's information leakage from the website traffic variable to the dependent variable that we're trying to predict. This problem also occurs with variables like branded search and affiliate programs, which are both very tightly related to the dependent variable we care about. If you're including branded search spend in your holdout forecast, you're likely helping your model to cheat another source of information. Leakage can occur in multi stage models where only one stage is held out, but the other stages aren't. Facebook's open source Robin package actually suffers from this problem. Its validation accuracy doesn't reflect a true holdout forecast since the holdout data are included in the first stage model that estimates the baseline and time series components of the model via another tool called Prophet. This allows information to leak in from the dependent variable. So it's very difficult to get a sense for Robin's true holdout forecast forecast accuracy. So once you have a system for holdout forecast accuracy measurement and backtesting, now you just need to make it continuous. This continuous measurement will help give you and your stakeholders confidence that when you're making a recommendation today about how they should change their budget next month, you will not be leading them astray. This is because you've already shown them that you can do this consistently over and over and over again with this backtesting system. As we've said before, model validation is all about continuously validating the MMM results from multiple different angles. In the remainder of this series, we'll be talking about other methods for doing MMM model validation, so make sure you watch those as well. We'll see you in the next episode.
The Continuous Testing Loop That Separates Great Marketing Teams From Average Ones
