Michael Kaminski explains how to validate marketing mix models using parameter recovery simulations. The technique tests whether your model can accurately recover the true parameters you built into synthetic data, catching errors before you trust real results.
Statistical modeling is hard. There are millions of ways for the model to go wrong and only one way for it to go right. How do we know if the model we're using is appropriate for the question we're trying to answer? How do we validate if the ground truth is unknown? I'm Michael Kaminski, an econometrician entrepreneur and marketing science researcher. Welcome to Modern analytics for marketers, our YouTube series that helps marketers learn and put into practice modern marketing measurement frameworks. This is the fourth episode in our series on how to validate marketing mix models. Today we're going to talk about parameter recovery, which is an incredibly important technique for all types of statistical modeling problems. If you're doing any type of statistical inference work, parameter recovery should always be your first step so that you know where your method works and where it doesn't. As we've discussed in the previous videos in this series, one of the things that makes marketing mixed modeling really challenging. Challenging is that the ground truth that we care about, the true incrementality of our marketing channels is unobserved. It's unknown and unknowable. So we are trying to build a statistical model where it's impossible to know if we've gotten the correct answer, right? Well, not exactly. What if we did know what the right answer was? Then we could run our model and compare the model's results to the truth and then see how well we did. That'd be helpful, right? Well, that's what we can do with a parameter recovery exercise. The idea is that we can generate fake data where we, the modeler, know exactly what the truth is, and then we can test our model on the generated data set to see if it can correctly find the truth. When we know that truth, you might be thinking, wait, that's too easy. If I already know the answer, it should be easy to build a model that can find that answer. Well, sort of. A few points on that. First, you want to use this parameter recovery exercise as a test of a fully automated system. That is, you want to randomly generate new data and then run your fully automated code or modeling software against those randomly generated data to see if you can find the truth without human intervention. You want to actually run this parameter recovery exercise hundreds or thousands of times to see how well your model does across many different runs so you aren't just getting lucky on one run. Second, if your model always recovers the parameters you care about, you probably aren't being creative enough. Every statistical model must make simplifying assumptions. And so the point of the parameter recovery exercise is to understand under what circumstances your model works well and under what circumstances it doesn't. You should be able to come up with situations where your model can't correctly recover the parameters. If you aren't able to do that, you aren't being creative enough. So try harder. Come up with different, more difficult situations where your model doesn't work as well. So what does this parameter recovery process look like in practice for marketing mixed models? First, start with research. You should be talking to marketers, marketing scientists, reviewing the literature to try to understand what the real world actually looks like. How does marketing work in the real world? Second, you need to generate simulated data based on the research you did in step one. Remember, the goal is to try to generate data that are as realistic as possible, not just data that you think will be easy to model. Finally, you should build a system that makes it easy to randomly generate data and then test your modeling approach against those data. You should be able to run this program tens, hundreds or thousands of times so that you can understand how often your model gets to the right answer. So that's the idea. This technique can and should be used for any any type of inferential statistical modeling that you're doing. But what features should you include in your data generating code for marketing mixed modeling Specifically, here's a non exhaustive list of things that you should consider. 1. Does marketing channel effectiveness change over time by much and how fast? 2. Can we generate data with time shifts and diminishing marginal returns curves that are unknown to the modeler? How does that impact our results? 3. What's the relationship between seasonality and marketing effectiveness? Are they independent or do they interact? 4. Are base sales fixed over time or do they evolve over time with seasonality or trend? 5. Do promotional events have all of their effect on one day or are they spread out over time? Are there pull forward or pull backward effects? And then 6. Are channels truly independent or are there relationships between the channels? Does activity on TV Drive Branded Search? This is just a non exhaustive list to get you started, but hopefully this is helpful for thinking about how you can apply this parameter recovery technique to your own model. In the next episode, we'll be covering stability and robustness checks for marketing mixed models, so make sure to like this video and subscribe to our channel so that you don't miss it. In the meantime, good luck modeling.
