How to Build a Media Mix Model (MMM) | Complete Tutorial by Recast

Have you ever wondered what makes the difference between a good and bad statistical model, or how to avoid the pitfalls that most modelers fall into when building statistical models for real-world problems? You're in the right place. I'm Tom Vladek, statistician, marketing scientist, and co-founder of Recast. I'm here to show you the secrets and science behind building complex and reliable models in the context of media mix models built on top of Recast. This is How to Build a Media Mix Model That Won't Destroy Your Marketing Program, presented by Recast, Episode 1. To start things off, let's face the facts. Media mix modeling is hard. It's really easy to throw your data into an algorithm and get results back, but more often than not, this will lead you astray. We've seen it hundreds of times. A brand wants to try this hot new thing and believes they can just dump their data into a black box algorithm or simple regression and get results on the other side. The fact is, MMM is incredibly challenging. We like to say it's harder to imagine a tougher statistical problem than MMM. It's a small data, low signal problem where millions of dollars are on the line. If you've been tasked with building an MMM and want to generate results that are actually useful, you need to take the building process seriously. For context, at Recast, we've built an MMM SaaS platform that helps brands develop, deploy, and use media mix models efficiently. We have spent a ton of time thinking about how to automate and develop the best practices for developing a media mix model, and we want to share those findings with you. In the past, MMM modelers would do a ton of offline analysis in R scripts or Jupyter Notebooks and then eventually they'd compile their results into a PowerPoint deck and share those out. But here's the thing: if you're only building a one-off MMM, it's really easy to bake in tons of cheats where you think you're generating sensible and actionable results, but really you've walked down the wrong path. We've heard of people doing really indefensible practices like freezing coefficients. This would mean presenting coefficients from two models as if they came from one. Or even simply overwriting results they didn't like. Since Recast is designed to be efficient and automated as well as ethical, those practices simply wouldn't work for us. So we had to take the state of the art in statistical model development and apply it to MMM. In this video series, we're going to walk you through our model building best practices that brands have used to get the most out of their MMM and avoid the pitfalls that many brands fall into when building an MMM and using it. Here's what we're going to cover in the following episodes. First, choosing or developing a model. Second, model configuration or prior setting based on background and expert knowledge. Third, the configuration compatibility check. Fourth, the parameter recovery check. Fifth, the stability and robustness checks. And finally, 6, the out-of-sample predictive accuracy. We'll dedicate a video lesson to each of these steps so that you can follow along as you're building your own models. Now remember, we'll be explaining these in the context of the Recast platform, but each of these is generally applicable to statistical models in general, so you can use them for your own work. You'll just have to write a bit of code.

Where do statistical models come from? Are they flown in by storks? What is so magic about linear regression that makes it so commonly used? If you've ever wondered what it truly means to build a statistical model or how structural model assumptions are made, then you're in the right place. I'm Tom Vladek, statistician, marketing scientist, and co-founder of Recast, and I'm here to show you the secrets and science behind building complex statistical models in the context of the marketing mix models that we build here at Recast. This is how to build a media mix model that won't destroy your marketing program, presented by Recast, Episode 2. Media mix modeling is nothing new. Businesses have been using media mix modeling to understand the performance of their business for a long time. That means that there's academic literature and a wide range of studies to build off of. Unfortunately, many people are still using modeling approaches that we think are totally backwards. So this backwards way of working is starting with a model and then working backwards to make your assumptions fit the assumptions of the model. There is a better way. It starts by working from your assumptions and core beliefs and then finding or building a model that handles those assumptions and beliefs natively. If you started with linear regression and worked backwards to make your assumptions and data fit the requirements of a simple linear regression, you would get a media mix model that didn't perform very well. The approach that we recommend generally comes from the research and the literature on causal inference. It starts by creating a map of how things fit together and what causes what. This is generally called a causal DAG. The DAG stands for Directed Acyclic Graph, but you can think of it as a visual way of describing how things fit together. For example, Consider how demand capture channels like retargeting or branded search work in the context of marketing. A really naive media mix model might show a marketing channel driving sales and an organic channel also driving sales. But what about retargeting and branded search? These are more complicated. Those channels themselves are also driven by marketing activity, and our DAG must become a bit more complicated to handle that. Here the DAG is showing how demand capture channels are influenced by demand generation channels, and both of them influence sales. Now, this DAG is still probably too simple. If you're developing one for your business, you need to account for things like seasonality, ad stocking, brand effects, exogenous variables, you name it. If you haven't created a DAG, we would really encourage you to do so. Now that you have your DAG, it's time to find a modeling framework that will allow you estimate the parameters of interest. It's important to recognize that in general, you're not going to be able to estimate everything that you want. You're going to have to make trade-offs in order to focus on the effects that are most important for your business. In general, in the context of MMs, the most important parameters are the incremental ROIs or CPAs for each of your marketing channels. So you're probably going to want to start there. If you are going to work with an open-source model, you'll want to make sure that you understand at a very detailed level how the model works under the hood, and that the assumptions that that model is making reflect your core beliefs as well. It's very likely that any open source model is going to not match your true beliefs in one way or another. Now, whether or not that's important will depend on what trade-offs you're making, what you really want to estimate, and what you're a little bit more flexible about. We generally believe that all of the currently available open source models are not sophisticated enough for most brands. So if you do decide to use an open source model, I would suggest considering that as the starting point on which you layer additional logic. At Recast, we've developed our own Bayesian model from scratch. A link to the documentation is in the description so you can read more about the assumptions that we've made. The DAG that we're working off is complex. Here is Antonio Vargas senior data scientist who's going to walk you through it.

I'm Antonio Vargas, head of the applied modeling team at Recast. The goal of Recast's DAG is to account for how marketing works. This is our representation of how different factors influence a brand's total sales or acquisitions through the mechanism of marketing. So conceptually, we break sales down into two main buckets. There's sales driven by paid media, and there's baseline sales, sales not driven by near-term marketing. So for example, on any given day, you will have people buying a cheeseburger at a fast food restaurant because they've seen an ad for the restaurant recently, and you'll also have people buying a cheeseburger there who have not seen an ad recently. So for each of these two things, sales driven by paid media and baseline sales, we can break them down further. We can talk about what directly or indirectly influences those sales. So sales driven by paid media is directly influenced by the effectiveness or quality of the marketing and also the amount of spend either on demand generation media or on demand capture media. So these are media targeted at different points in the conversion funnel. On the other side, sales driven by paid media are indirectly influenced by things like brand, seasonality, promotional activities, and all kinds of other exogenous factors. Things get a lot more complicated than this in practice, but these are some of the broad assumptions that inform Recast's DAG.

Part of what Antonio is explaining here is how seasonality impacts both organic sales and the efficacy of demand generation sales at the same time. All of the underlying code that powers Recast is driven by the causal model that Antonio has just described. So what's the most important thing to take away? Whether you're developing a model in-house based off of an open source model or working with another vendor, you should understand the causal model and how it relates to the model code that is being run to estimate your MMM.

You can't just take any open source package, run data through it, and expect to get good results on the other side. This is especially true in the world of media mix modeling where causal inference matters. In this video, we're going to talk about media mix model configuration and hyperparameter setting in order to get valid causal inferences. I'm Tom Vladek, statistician, marketing scientist, and co-founder of Recast. I'm here to show you the secrets and science behind building complex, reliable models in the context of the media mix models we build at Recast. This is How to Build a Media Mix Model That Won't Destroy Your Marketing Program, presented by Recast, Episode 3. Remember, no statistical models are assumption-free. Either you hide the assumptions or you make them explicit, but there is no escaping the assumptions. In the case of a media mix model, which is naturally very complex, you'll need to make a lot of assumptions and configure the model in a way that matches your business and the questions you're interested in answering. This configuration includes many things. It might include things like hyperparameters in the model or priors if you're in a Bayesian setting, but it also includes things like which variables do we include in the model and what transformations do we apply to the data before it goes in the model at all. One important thing, it's important to do this configuration before the model actually sees the data. We want to start with our assumptions and only then run the data through the model. The problem is, once you see the results, you might be incentivized to go back and tweak your assumptions to make the results look a little bit different. But this is bad science, and it can lead to biased models that will lead you and your company astray. There's a reason that Bayesians call these kinds of hyperparameters priors, because they're supposed to be the the state of the world, your beliefs, the state of the model before it sees the data. Setting hyperparameters and priors for a complex media model is not easy. In general, if you're following best practices, you want priors that are going to be as uninformative as possible, meaning that they should rule out impossible combinations of the data or things that you know a priori cannot be true, but other than that, should not impact the results too much. One important question is, how do you even get the priors to begin with? To elicit priors from the marketers that are setting up Recast, we have a set of questions that we ask that can get their beliefs about how marketing is performing, and that can be translated into priors that we set into the model directly. This is how the model is customized for each customer and the data that they have. Here's Andy Bennington, head of strategic insights at Recast. Walking through a few of these questions.

Every Recast model build process begins with a deep dive into the client's business. This deep dive ultimately boils down to just a few key questions. Now, before I get into those questions though, it's important to note that for all of them, we don't expect our clients to have the precise answers to them because if they did, they wouldn't need to be talking to Recast. Instead, we're asking these questions so we can set those boundaries for the model that reflect the uncertainty that our clients have about their own business. All right, let's get into the questions. Question number 1 is how much of your business would remain if you turned off all of your marketing tomorrow? Now, this is a very complicated question, but some of the factors we encourage clients to think about when answering are how strong is your brand compared to your competitors? Do you have an obvious distribution advantage or disadvantage in your category? And just in general, how competitive is your industry? Remember, we're trying to cast a wide net here so that Recast can go and do the work from there. Question number 2 is how long did it take your customers to go through that decision journey when they're making a purchase and which phase of the journey are each of your marketing channels allocated towards impacting. For example, some channels might be targeted towards earlier stages of that decision-making process. So we want to tell the model to look for their impacts further away into the future from when you spent those dollars. The other end of that spectrum, though, is some channels might be targeted towards getting people right across the finish line. Maybe paid search is a good example here. And we want to tell the model to look for the impact within a few days of someone clicking on a paid search ad. It wouldn't make sense for the model to come back and say that paid search has this impact a year later. Question 3 is, well, outside of marketing, what other factors impact your business? And this question is admittedly broad, but we break it down into more digestible chunks like major promotions you've run in the past, some macroeconomic factors that might impact your business, and business model changes or shifts to the way you've actually structured The way clients move through your website or your conversion flow, those types of things wouldn't otherwise be captured within your marketing but could have a large step change in the impact of your revenue. Ultimately, the goal of all these questions is to bring the client an actionable, rigorous model that can drive as much value as possible from day one. All right. So those are the main 3 questions that we'll talk about when we're setting up a Recast model to make sure we're giving you a customized, actionable, and most importantly, a very rigorous model for you to drive as much value with Recast as possible. From day one.

But if you're not using Recast, here are a few things that you want to make sure you don't do during this prior setting process. First of all, you don't want to just input data from multi-touch attribution or surveys or ad platforms. These methods are all biased and you don't want to make sure that their biases flow through into the model. Second, you don't want to just assume that what was true in the past is true today without verifying it first. We often see this as a source of confusion where marketing teams might not have thought enough about how things might have changed with respect to their performance since some exercise many years ago and how things might be different today. That's very relevant to this modeling exercise. At Recast, we like and generally recommend setting broadly uninformative priors that don't bias the model too much. However, it's always important to write down the priors and to present the prior results along with the model results so that any biases in the priors are visible. We think that transparency is incredibly important and good modelers should be transparent about the assumptions being made, whether via priors, hyperparameters, or model structure. And so all of these things should be transparently available to anybody that is using the model. So in general, the broad principles for hyperparameter setting are One, start with the most important business questions to answer. Two, then think about the relevant assumptions that you're willing or not to make within the relevant modeling framework. Three, try to avoid biasing the model with priors or hyperparameters wherever possible. Four, be transparent about the assumptions being made whenever communicating results. These are a good start to getting valid causal inferences out of your media mix model. In the next episode, we'll talk about doing a configuration compatibility check as a way to make sure your assumptions are at least somewhat plausible. Media mix models are complex and there are a ton of hyperparameters that we have to set before we can even start training the model. How can you know if you're right or even close to right? I'm Tom Vladek, statistician, marketing scientist, co-founder of Recast. I'm here to show you the secrets and science behind building complex reliable statistical models in the context of the media mix models built on top of Recast. This is how to build a media mix model that won't destroy your marketing program, presented by Recast, episode 4. The first thing that we want to do with our model is verify that our hyperparameters are at least compatible with the data and that it's at least possible to yield reliable results. In Bayesian models, this works via simulation. We feed the data into the model, but we don't let the model actually learn anything. We do a prior-only run and then look at what comes out the other side. It's essentially a model only with the priors, learning nothing from the data. What we're doing here is we're checking that the configuration along with the priors simulates a KPI that is at least reasonable within the bounds of what a business that we're modeling could potentially be producing.

Right.

So for example, if we're modeling a business that does hundreds of millions of revenue every year and our priors would simulate revenue in the ones of millions, that would be a problem. And vice versa is true too. If we're modeling a business that does single-digit millions of revenues and the priors simulate revenue in the hundreds of millions, we have a problem. Not my area of expertise, but the same principle should be possible in a non-Bayesian framework. Although you might have to do some custom modeling in order to get the model to actually generate results if it's not a generative model to begin with. This step can help quickly identify problems with the configuration, even if it's just a typo or something like that. Though they can also help identify when a brand's beliefs about its marketing program are just incompatible with the data. For example, they might think that their intercept or where their baseline is 50% of sales, but their advertising also has a 5x ROI. It may be the case that when you add 5x times spend on top of the 50% that's organic, you get a result that's like 3 times their actual sales. Both things can't be true. Let's take a look at what the output of this check looks like. Here we can see what the range of simulations coming from the prior look like in blue and the actuals just plotted against the data. Remember, the data didn't learn anything from the actuals. It's just being plotted against this data. It's in black. The good news in this case is that the simulations in blue are in the same order of magnitude as the actuals in black, and that the black kind of goes through the middle of the blue, meaning that our priors are producing results that are roughly the same order of magnitude in the ballpark of the actual data. There doesn't seem to be anything screamingly wrong here. Now sometimes there are. For example, we might see that the actual data has massive spikes in the dependent variable that aren't captured by the prior simulations. Now this means that our model is not really gonna be able to account for those massive spikes. And so there's something wrong in our configuration that we'll need to adjust in order to get the model to be able to explain those large spikes in whether it's revenue or new customer acquisition before we actually start modeling. Or it could be the case that our priors are consistent with the most recent data. But as we look into the past, something about the priors just is totally inconsistent with data one year ago or two years ago. These are all things that we'd want to observe see in a plot, understand, and address before we start to do any real modeling. Media mix models need to be actionable for marketers. If a model produces wild swings in its outputs every time it's updated, that indicates that there's something seriously wrong with the model and that it should not be trusted for decision-making. That's why model stability is so important. I'm Tom Vladek. Statistician, marketing scientist, and co-founder of Recast. I'm here to show you the secrets and science behind building complex, reliable models in the context of the media mix models that are built on top of Recast. This is How to Build a Media Mix Model That Won't Destroy Your Marketing Program, presented by Recast, Episode 5. Let's take a step back. The goal of this series is to chart a path towards a model building process that yields statistically reliable and scalable media mix models. So far we've talked about model selection and development, configuration compatibility checks, and parameter recovery checks. Now let's talk about one of the most critical steps in the build process, stability and robustness checks. Many modelers working on building media mix models are familiar with the problem of robustness. These modelers put in a ton of time building media mix models that look good, but then at the last minute they need to change an assumption or slightly change the underlying data, and all of a sudden everything falls apart like a house of cards. All of the results change and all of their hard work goes to waste. When this happens, it means that the model is not robust. This is bad. In general, models that are not robust are not reliable. Statistical models that are picking up real signal in the data tend to be quite robust. Basically, the story remains the same no matter how the data is sliced or diced, or if small assumptions in the model are tweaked. When model results change dramatically with small changes to the assumptions or the underlying data, it means that the model does not work and the results cannot be relied upon for decision-making. If every time you update your Media Mix model you get different results, it means that you cannot tell fact from fiction, and it's highly likely that following the model's recommendations is going to drive a loss instead of driving profit. This is a small aside that's indicative of a much larger problem in the world of marketing measurement. Media mix models must be stable and robust to new data in order to be actionable. How can we know if we have a robust model? First, let's talk about measuring stability before we start assessing it in our models. Here's Chelsea Parlett, consulting statistician at Recast, with more.

Useful media mix models don't just give you point estimates for parameters as output, they also give you the model's uncertainty. Uncertainty around those estimates. While your model's point estimates for parameters might vary week to week, the uncertainty interval given by your model should be fairly similar since that interval should cover the range of parameter values that are consistent with the data. So for a single model parameter like the ROI of non-brand search, stability can be measured as the amount of overlap between the uncertainty interval for this parameter from your previous model run and the uncertainty interval for this parameter from your latest model run. When comparing two models, we want to do this overlap percentage calculation for every parameter in the model and build that up into an overall stability calculation for the model. One small note: when doing this overall stability calculation, you may want to account for the differing impact different parameters can have on your model's dependent variable. The bigger the impact a parameter can have on your model's predictions, the more concerned we are with it being stable.

While the level of stability you require is dependent on your business and how you'll use the MMM to make decisions, there are correct and incorrect ways to go about assessing it. The wrong but easy way to achieve stability would just be to force every model run to be very similar to each other, either by priors or overriding results. Methods like this, like freezing coefficients, are cheating. And mean that this check is no longer really assessing the model's robustness. A similarly flawed practice to look out for are practitioners that achieve stability simply by not refreshing their model very frequently. Both of these practices are still unfortunately very common today. So what's a better way to assess model stability? At Recast, we use a test called Stability loops. These mimic a weekly refresh cadence by looking at multiple models, each with one additional week of data, and quantifying how similar their results are. What we're looking for is that the model is learning from new data, but that one week of new data out of about 100 cumulative weeks doesn't cause huge changes to the model's estimates. Stable estimates week to week are evidence that our model is one, 1, picking up on a consistent signal from the data rather than overfitting to noise, and 2, is robust to seeing slightly different time ranges. Even if you are only refreshing the model once a month or even once a quarter, you should still run these robustness checks in order to validate that the model's results are robust to small changes in the underlying data. At Recast, we think that weekly refreshes are an excellent way to prove that the model is robust since the robustness check is effectively always on. When somebody is only refreshing their MMM once per quarter or once per year, it's easy for them to cheat and hide robustness checks simply by running different variations of the MMM until the models look similar enough and they look right. Instead, we believe that models should constantly be validated for robustness and predictive accuracy.

Medium mix models are powerful observational models. This means they're subject to overfitting and can easily lead to correlation versus causation confusion. Out-of-sample predictive accuracy checks means checking how well a model can predict the KPI of interest on data that it hasn't seen before. This check is a critical step in identifying problems with your model. And while this check can't prove that a model has identified the true underlying causal structure structure in a dataset, this check can provide evidence for or against that claim. Out-of-sample accuracy checks are also one of the best ways to validate a model and then start building trust in its forecasting ability. In this video, we're going to explain what out-of-sample predictive accuracy checks are, why they're so important for media mix modeling, and how to really run them properly. I'm Michael Kaminsky, statistician, marketing scientist, and co-founder of Recast. I'm here to show you the secrets and science behind building complex, reliable models in the context of the media mix models that are actually built on top of Recast. This is How to Build a Media Mix Model That Won't Destroy Your Marketing Program, presented by Recast, Episode 6. Broadly, our goal is to define a model building process that produces robust, statistically rigorous media mix models. So far, we've discussed how to select and configure an initial model, how to run parameter recovery checks on that model, and how to test for its stability and robustness. The last step in our model building process is to check the out-of-sample predictive accuracy of the model itself. Before we get started, we have to recognize that in the context of media mix modeling, we can't use the same off-the-shelf cross-validation tools that you might be familiar with from a machine learning context. The reason why is because in the world of marketing, everything is highly time dependent. So individual observations like days or weeks aren't actually independent independent from each other. If I have the data on what happened on Tuesday and Thursday, it's not actually very difficult to predict what must have happened in between those days on Wednesday. So when validating media mix models, it's critical that we're holding out a full time series of data in the future with respect to the data cutoff date. This sounds complicated, but the idea is actually pretty simple. We want to chop off the last n days of data, say the last 30, 60, or 90 days, and then train the full model up to that point. Then we ask the model to predict what will happen in the data that it hasn't seen before. We then compare the model's estimates of predictions with the actuals over that time period to see the model's out-of-sample predictive accuracy. This is an extremely important check in the model building process, but it can be a little difficult to get right because you need to be careful to make sure that there isn't data leakage between the model training dataset and the model outcome dataset. This can be more subtle than you might expect. For example, if you are training your model on the full dataset to generate an estimate of seasonality and then you use that in the model, well then you haven't really done the full holdout check because you estimated the seasonality on the full dataset and then you use that in the model training process. And so there was data leakage or information leakage between the training process and the validation check. And so when you're doing these checks, it's really important to make sure that you design the process to avoid any information leakage between the training process and the validation step. If the model fails to capture major trends in future data, it probably doesn't have a good read on what was actually driving past performance. In practice, you should run and review many out-of-sample predictive accuracy checks of differing lengths of time to build confidence in a model's forecasting ability. And if you update your model frequently, you can automate these checks with each model refresh. And that's exactly how we do things at Recast. When Recast models update every week, a new series of out-of-sample predictive accuracy checks are triggered on a brand's backlog of old and pre-trained models. This creates an out-of-sample forecast accuracy report card that looks something like this. So here we see for the 7-day-old Recast model, the platform runs a brand's actual spin data through the model and has it make a forecast. It then compares the model's forecasted outcome to the known outcome over the last 7 days and reports on that model's forecast error. And then we repeat the same process for models that are 30, 60, and 90 days old to assemble the full out-of-sample accuracy scorecard. Automated checks like this are important for a few reasons. If forecasts are off, this can signal issues in the underlying model structure and the need to make changes to a model's configuration. If forecasts are consistently strong, this can give a brand confidence that their models have identified true causal signal and can be trusted for decision-making. One of the most common mistakes we see is when a brand or modeler just inherently trusts the output of a media mix model. We think you should do exactly the opposite. You should be skeptical of any model by default and use a process of ongoing validation to build trust in the model over time. Out-of-sample predictive accuracy checks and lift tests are critical in this validation process. As we wrap up this series, it's important to remember that running data through a media mix model and getting something out on the other side, that's the easy part. The hard part is really doing it correctly, following a rigorous model building process, holding your model to high performance standards so that when you actually use it to make decisions, you don't ruin your marketing program. But if you do this whole process, you'll be rewarded with a robust robust, accurate model that your marketing team can actually trust.