How to do configuration compatibility checks in MMM | How to Build an MMM Ep.4

Media mix models are complex and there are a ton of hyperparameters that we have to set before we can even start training the model. How can you know if you're right or even close to right? I'm Tom Vladek, statistician, marketing scientist, co founder of Recast. I'm here to show you the secrets and science behind building complex, reliable statistical models in the context of the media mix models built on top of Recast. This is how to build a media mix model that won't destroy your marketing program or Presented by Recast. Episode 4 the first thing that we want to do with our model is verify that our hyperparameters are at least compatible with the data and that it's at least possible to yield reliable results. In Bayesian models, this works via simulation. We feed the data into the model, but we don't let the model actually learn anything. We do a prior only run and then look at what comes out the other side. It's essentially a model, only with the priors learning nothing from the data. What we're doing here is we're checking that the configuration along with the priors simulates a KPI that is at least reasonable within the bounds of what a business that we're modeling could potentially be producing. Right? So for example, if we're modeling a business that does hundreds of millions of revenue every year, and our priors would simulate revenue in the ones of millions, that would be a problem. And vice versa is true too. If we're modeling a business that does single digit millions of revenues and the priors simulate revenue in the hundreds of millions, we have a problem. Not my area of expertise, but this same principle should be possible in a non Bayesian framework. Although you might have to do some custom modeling in order to get the model to actually generate results. If it's not a generative model to begin with, this step can help quickly identify problems with the configuration, even if it's just a typo or something like that. Though they can also help identify when a brand's beliefs about its marketing program are just incompatible with the data. For example, they might think that their intercept or their baseline is 50% of sales, but their advertising also has a 5x ROI. It may be the case that when you add 5x times spend on top of the 50% that's organic, you get a result that's like three times their actual sales. Both things can't be true. Let's take a look at what the output of this check looks like. Here we can see what the range of simulations coming from the prior look like in blue, and the actuals just plotted against the data. Remember, the data didn't learn anything from the actuals. It's just being plotted against this data. It's in black. The good news in this case is that the simulations in blue are in the same order of magnitude as the actuals in black, and that the black kind of goes through the middle of the blue, meaning that our priors are producing results that are roughly the same order of magnitude in the ballpark of the actual data. There doesn't seem to be anything screamingly wrong here. Now, sometimes there are. For example, we might see that the actual data has massive spikes in the dependent variable that aren't captured by the prior simulations. Now, this means that our model is not really going to be able to account for those massive spikes. And so there's something wrong in our configuration that we'll need to adjust in order to get the model to be able to explain those large spikes in whether it's revenue or new customer acquisition before we actually start modeling. Or it could be the case that our priors are consistent with the most recent data. But as we look into the past, something about the priors just is totally inconsistent with data one year ago or two years ago. These are all things that we'd want to observe, see in a plot, understand, and address before we start to do any real modeling.