MMM validation and common modeling pitfalls to avoid | Modern Analytics Ep. 2

Marketing mix modeling is so hot right now, but there's one critical piece missing from all of the guides and tutorials you can find online. How to actually know if you've built a good model welcome to Modern analytics for marketers, our YouTube series that helps marketers like you learn and put into practice modern marketing measurement frameworks. This is the second episode in our series on marketing mix modeling, sometimes known as just MMM, and in particular on how to validate MMMs. Today we'll talk about the most important topic in MMM that the normal tutorials won't teach you. Validation. Why is validation important? Well, it allows us to prove to ourselves and others that our model can be trusted. In order to make real marketing budget optimizations with confidence, we need to trust the output of our model. I'm Michael Kaminski, econometrician, entrepreneur and marketing science researcher. Welcome to Modern analytics for Marketers presented by Recast. If you watched our previous episode, you know that MMM is a powerful tool for marketing measurement, planning and forecasting. But as we all know, with great power comes great responsibility. It's our responsibility as marketing scientists not just to build a model, but to build a model that can be trusted and actually used. It's really easy to build a bad marketing mix model and incredibly difficult to build a good one. There are hundreds of videos on YouTube and tutorials that will show you how to build a model. Unfortunately, what those videos won't show you is how to build a good marketing mix model and then how to validate it. When you're building an mmm, you have to remember there are millions of ways for the model to be wrong and only one way for it to be right. In the remainder of this video you can we'll talk about one why the problem of validation is so difficult in the context of MMM, 2 the problems of overfitting and model misspecification, 3 why validation requires multiple different approaches and angles and 4 the most important methods for validating MMMs. First, why is the problem of validation so difficult in the context of marketing mixed models MMMs are fundamentally trying to solve a causal inference problem. We want to understand how different changes that we might make to our marketing budgets will change our business performance. This isn't simply a prediction problem, but rather an attempt to understand the true causal relationships between our marketing activity and our business outcomes. Validating causal inference models is much, much, much more difficult than validating a simple prediction only model, and so we need a different toolset and approach to validating this type of model. The fundamental problem is that the thing we care about as researchers, the true incremental impact of an additional dollar spent on some marketing channel. It's unknown and unknowable. No one knows. No one can know the true value of an additional dollar spent on Meta or on tv. There is no fundamental law of marketing physics or nature to fall back on, and there's no way to ask people or track them sufficiently well to know what that true impact is. So our job as modelers is to try and validate what we've learned from our model without being able to know what the true answer really is. This is the fundamental problem of validating MMMs beyond just the basics of doing causal inference. The MMM problem is compounded because things change over time. What might have been true six months ago about marketing performance might no longer be true today. Thus, the problem of model validation in MMM is a problem that not only needs to be solved once, but it's actually a problem that needs to be continually addressed over and over and over again. Now let's talk about the problems of overfitting and model misspecification. MMMs are powerful models, and that means they're subject to what modelers call overfitting. The idea is that you can build a model that fits really, really well to the data that the model is trained on, but that hasn't actually found the underlying causal relationships that we care about. Overfitting happens when a model is too powerful and fits to the noise in the data instead of the signal. And if your model's too overfit, it will be fit only to noise and will miss out on the signal entirely. Normal methods for evaluating model fit are often the cause of overfitting. If you look at your in sample R squared or MAPE or RMSE metrics, these metrics will go up as you add more variables and features to your model. Unfortunately, these metrics are leading you astray because you're just overfitting your model to the data to the noise in the data. These metrics, mape, rmse, R squared, all in sample, they'll look amazing. But the model will not have found the true underlying relationships and instead will just be perfectly matched to the random noise in the data that we happen to be looking at. What that means in practice is that the results we get from the model will be wrong. They will not be driven by the true causal signal in the data and instead just by the noise. Then when we go to actually use the model to make budget changes, we could end up costing our business millions of dollars. This is bad. Another related problem is model misspecification. Many MMM tutorials and even models built by expensive consulting firms use a standard linear additive regression to fit the model. Linear regression is a pretty great powerful tool for doing statistical modeling, but it implicitly makes a number of assumptions about how marketing works. If the way marketing actually works doesn't match the assumptions of your model, then your model is misspecified. That is the specification of the model doesn't match the real world. There are many good examples of this. Most marketers believe that marketing effectiveness of a channel can change over time depending on things like creative or targeting strategy, competitor activity, global pandemic. But then they use a modeling framework that assumes that marketing performance is fixed over time like a standard linear regression. Similarly, most marketers believe that marketing effectiveness is influenced by seasonality. Your marketing gets more effective in the summer than in the winter if you're selling something like sunscreen. But then the modeler will make the assumption that those two factors are totally independent, which can lead the modeler to get exactly the wrong answer when they go to run the model. Overfitting and model misspecification are both big problems, but it's not always easy to tell if they're problems for your model. The popular open source packages don't make it easy to check for these issues, and neither do out of the box model fit statistics that you might have learned about in your college stats class. So what can we do about it? This is where validation comes in. Validation will help us detect problems with overfitting, model misspecification, and other common modeling problems that you might encounter. In order to get model validation right, we'll need to approach the problem from multiple different angles. Remember, the truth is unknown and unknowable, so we need to validate our inferences with multiple different strategies in order to home in on the truth. The most important methods for model validation are 1 holdout forecast accuracy and backtesting 2 parameter recovery 3 stability and robustness checks 4 lift tests and experimentation and then 5 dynamic spin deployment and forecast reconciliation. These methods help us to validate our MMM model, and they're critical techniques that every modeler should know. We'll dedicate a future episode in this YouTube series to each of those validation methods so that you can apply them to your own modeling practice. Our next episode will be on holdout accuracy and backtesting so that you can forecast with confidence. Make sure that you subscribe to our channel. Subscribe you don't miss it. And if you have any questions around validation, please leave the question in the comments. Thank you for watching and we will see you in the next episode.