How to do MMM stability and robustness checks | Modern Analytics Ep. 5

Personally, I hate when people say that statistical modeling is more art than science. It's generally something that someone says right before they're about to do some very, very poor science. A good marketing mixed model should be all science. As modelers, it's our job to deeply understand and communicate the limitations of our statistical modeling efforts. Doing good science is hard, but if we do it well, we have a shot at learning fundamental truths about our business. If we do it poorly, we're likely to lead ourselves and our business astray. I'm Michael Kaminsk, an econometrician, entrepreneur and marketing science researcher. Welcome to Modern analytics for marketers, our YouTube series that helps marketers like you learn and put into practice modern marketing measurement frameworks. This is the fifth episode in our series on how to validate marketing mix models. Today we're going to talk about stability and robustness checks, two very important techniques for validating marketing mix models in the field of statistical modeling. Robustness refers to the idea that the results of an analysis shouldn't change when the inputs are tweaked just a little bit, or when the assumptions are tweaked just a little bit. When the results that we care about are robust, it gives us evidence that our statistical model has found some underlying truth about the world and isn't just fit to noise in the data. So how do we check for robustness in practice? What are we looking for? There are three different types of robustness checks that we recommend for a marketing mix model. 1. Varying the underlying dataset being modeled by running the same model on different subsets of the data to see how the results change. 2. Varying your model's structure to see how sensitive the results are to different structures, including which variables are controlled for in the model. 3. Varying your modeling assumptions to see how different assumptions that you make during the modeling process impact the final results. So let's talk through what this actually looks like in practice. In the first robustness check, we want to vary the underlying data that we're feeding into the mmm. What if we exclude the last month of data? What if we exclude the first month of data? In the context of a marketing mix model, we generally care most about the effectiveness estimates for each of the marketing channels we're operating in. So, like the incrementality of each marketing channel. So what we're looking for in this robustness test is do the estimates for channel effectiveness stay the same or do they change when we make those small changes to the underlying data, like removing the last one, two, three, or four weeks of data. If the estimates from the model jump around substantially, then that means that our results are not robust and it often indicates that the statistical model itself is misspecified. In the second robustness check, we want to see how robust our results are to changes in the model structure itself. In particular, which variables are being controlled for in the model. One place that we've seen many MMM analysts go wrong is that they include lots of different control variables in the model and then just sort of randomly add or subtract different control variables until they get the answer that they want. This is very bad practice. As part of our robustness checks, we want to understand how our model structure is impacts the results that we see. In particular, we want to know how our effectiveness results change with different subsets of control variables. If we do or do not control for inflation, how much does that impact our estimate for TV effectiveness? This exercise should be repeated for all of the various control variables that an analyst might be choosing to include or not include in the model. The last robustness check involves testing how sensitive our results are to assumptions that we modeler might make. In many MMM frameworks, the modeler is choosing values like ad stock rates or diminishing marginal returns curves for each channel. This means that you should test how sensitive the final results are to those different assumptions. For example, what if you use an ad stock rate of 0.5 instead of 0.8 for the TV variable? How much does that change your final results? You should be reporting out all of these different variables and how much changes to assumptions and impact the final results that the marketer is seeing. These robustness checks are very important to doing good science. If your results are not robust to these different assumptions, you shouldn't hide it. It's critically important that the true range of uncertainty and instability in the results is accurately communicated. Otherwise, the next time you go to refresh the model, the results will change since they aren't robust. And you'll have to come up with an explanation as to why. In the next episode of this series, we're going to talk about how to validate an MMM model with lift tests and experiments. Make sure to like this video and subscribe to our channel so you don't miss it. Good luck modeling and we'll see you in the next one.