Tom Vladek and Chelsea Parlett show how to test whether your media mix model actually works. They cover stability checks, robustness validation, and the common mistakes that make models unreliable for real marketing decisions.
Media mix models need to be actionable for marketers. If a model produces wild swings in its outputs every time it's updated, that indicates that there's something seriously wrong with the model and that it should not be trusted for decision making. That's why model stability is so important. I'm Tom Vladek, statistician, marketing scientist and co founder of Recast. I'm here to show you the secrets and science behind building complex, reliable models in the context of the media mix models that are built on top of Recast. This is how to build a media mix model that won't destroy your marketing program. Presented by recast episode 5 let's take a Step Back the goal of this series is to chart a path towards a model building process that yields statistically reliable and scalable media mix models. So far we've talked about model selection and development, configuration compatibility checks, and parameter recovery checks. Now let's talk about one of the most critical steps in the build process, stability and robustness checks. Many modelers working on building media mix models are familiar with the problem of robustness. These modelers put in a ton of time building media mix models that look good, but then at the last minute they need to change an assumption or slightly change the underlying data and all of a sudden everything falls apart like a house of cards. All of the results change and all of their hard work goes to waste. When this happens, it means that the model is not robust. This is bad. In general, models that not robust are not reliable. Statistical models that are picking up real signal in the data tend to be quite robust. Basically, the story remains the same no matter how the data is sliced or diced, or if small assumptions in the model are tweaked. When model results change dramatically with small changes to the assumptions or the underlying data, it means that the model does not work and the results cannot be relied upon for decision making. If every time you update your mediamix model you get different results, it means that you cannot tell fact from fiction. And it's highly likely that following the model's recommendations is going to drive a loss instead of driving profit. This is a small aside that's indicative of a much larger problem in the world of marketing measurement media, mixed models must be stable and robust to new data in order to be actionable. How can we know if we have a robust model? First, let's talk about measuring stability before we start assessing it in our models. Here's Chelsea Parlett, consulting statistician at Recast.
With more useful media mix models. Don't just give you point estimates for parameters as output, they also give you the model's uncertainty around those estimates. While your model's point estimates for parameters might vary week to week, the uncertainty interval given by your model should be fairly similar since that interval should cover the range of parameter values that are consistent with the data. So for a single model parameter like the ROI of non brand search, stability can be measured as the amount of overlap between the uncertainty interval for this parameter from your previous model run and the uncertainty interval for this parameter from your latest model run. When comparing two models, we want to do this overlap percentage calculation for every parameter in the model and build that up into an overall stability calculation for the model. One small note when doing this overall stability calculation, you may want to account for the differing impact different parameters can have on your model's dependent variable. The bigger the impact a parameter can have on your model's predictions, the more concerned we are with it being stable.
While the level of stability you require is dependent on your business and how you'll use the MMM to make decisions, there are correct and incorrect ways to to go about assessing it. The wrong but easy way to achieve stability would just be to force every model run to be very similar to each other, either by priors or overriding results. Methods like this, like freezing coefficients, are cheating and mean that this check is no longer really assessing the model's robustness. A similarly flawed practice to look out for are practitioners that achieve stability simply by not refreshing their model very frequently. Both of these practices are still, unfortunately very common today. So what's a better way to assess model stability? At Recast, we use a test called stability loops. These mimic a weekly refresh cadence by looking at multiple models, each with one additional week of data and quantifying how similar their results are. What we're looking for is that the model is learning from new Data, but that one week of new data out of about 100 cumulative weeks doesn't cause huge changes to the model's estimates. Stable estimates week to week are evidence that our model is one picking up on a consistent signal from the data rather than overfitting to noise, and two is robust to seeing slightly different time ranges. Even if you are only refreshing the model once a month or even once a quarter, you should still run these robustness checks in order to validate that the model's results are robust to small changes in the underlying data. At Recast, we think that weekly refreshes are an excellent way to prove that the model is robust. Since the robustness check is effectively always on. When somebody is only refreshing their MMM once per quarter or once per year, it's easy for them to cheat and hide robustness checks simply by running different variations of the MMM until the models look similar enough and they look right. Instead, we believe that models should constantly be validated for robustness and predictive accuracy.
The Continuous Testing Loop That Separates Great Marketing Teams From Average Ones
