Michael Kaminsky explains how to test whether your media mix model actually works in the real world using out-of-sample checks. He walks through the methodology for validating your model's predictions against data it hasn't seen before, so you know your numbers are trustworthy before you act on them.
Media mixed models are powerful observational models. This means they're subject to overfitting and can easily lead to correlation versus causation confusion. Out of sample predictive accuracy checks means checking how well a model can predict the KPI of interest on data that it hasn't seen before. This check is a critical step in identifying problems with your model. And while this check can't prove that a model has identified the true underlying causal structure in a data set, this check can provide evidence for or against that claim. Out of sample accuracy checks are also one of the best ways to validate a model and then start building trust in its forecasting ability. In this video, we're going to explain what out of sample predictive accuracy checks are, why they're so important for media mix modeling, and how to really run them properly. I'm Michael Kaminski, statistician, marketing scientist, and co founder of Recast. I'm here to show you the secrets and science behind building complex, reliable models in the context of of the media mix models that are actually built on top of Recast. This is how to build a media mix model that won't destroy your marketing program. Presented by Recast Episode six Broadly, our goal is to define a model building process that produces robust, statistically rigorous media mix models. So far we've discussed how to select and configure an initial model, how to run parameter recovery checks on that model, and how to test for its stability and robustness. The last step in our model building process is to check the out of sample predictive accuracy of the model itself. Before we get started, we have to recognize that in the context of media mix modeling, we can't use the same off the shelf cross validation tools that you might be familiar with from a machine learning context. The reason why is because in the world of marketing, everything is highly time dependent, so individual observations like days or weeks aren't actually independent from each other. If I have the data on what happened on Tuesday and Thursday, it's not actually very difficult to predict what must have happened in between those days on Wednesday. So when validating media mixed models, it's critical that we're holding out a full time series of data in the future with respect to the data cutoff date. This sounds complicated, but the idea is actually pretty simple. We want to chop off the last n days of data, say the last 30, 60 or 90 days, and then train the full model up to that point. Then we ask the model to predict what will happen in the data that it hasn't seen before. We Then compare the model's estimates of predictions with with the actuals over that time period to see the model's out of sample predictive accuracy. This is an extremely important check in the model building process, but it can be a little difficult to get right because you need to be careful to make sure that there isn't data leakage between the model training data set and the model outcome data set. This can be more subtle than you might expect. For example, if you are training your model on the full data set to generate an estimate of seasonality and then you use that in the model, well then you haven't really done the full holdout check because you estimated the seasonality on the full data set. And then you use that in the model training process. And so there was data leakage or information leakage between the training process and the validation check. And so when you're doing these checks, it's really important to make sure that you design the process to avoid any information leakage between the training process and the validation step. If the model fails to capture major trends in future data, it probably doesn't have a good read on what was actually driving past performance. In practice, you should run and review many out of sample predictive accuracy checks of differing lengths of time to build confidence in a model's forecasting ability. And if you update your model frequently, you can automate these checks with each model refresh. And that's exactly how we do things at Recast. When recast models update every week, a new series of out of sample predictive accuracy checks are triggered on a brand's backlog of old and pre trained models. This creates an out of sample forecast accuracy report card that looks something like this. So here we see for the seven day old recast model, the platform runs a brand's actual spin data through the model and has it make a forecast. And then compares the model's forecasted outcome to the known outcome over the last seven days and reports on that model's forecast error. And then we repeat the same process for models that are 30, 60 and 90 days old to assemble the full out of sample accuracy scorecard. Automated checks like this are important for a few reasons. If forecasts are off, this can signal issues in the underlying model structure and the need to make changes to a model's configuration. If forecasts are consistently strong, this can give a brand confidence.
