MMM Configuration: Priors, variables, and best practices | How to Build an MMM Ep.3

You can't just take any open source package, run data through it and expect to get good results on the other side. This is especially true in the world of media mix modeling where causal inference matters. In this video, we're going to talk about media mix model configuration and hyperparameter setting in order to get valid causal inferences. I'm Tom Vladek, statistician, marketing scientist and co founder of Recast. I'm here to show you the secrets and science behind building complex, reliable models in the context of the media mix models we build at Recast. This is how to build a media mix model that won't destroy your marketing program. Presented by recast Episode 3 Remember, no statistical models are assumption free. Either you hide the assumptions or you make them explicit. But there is no escaping the assumptions. In the case of a media mix model, which is naturally very complex, you'll need to make a lot of assumptions and configure the model in a way that matches your business and the questions you're interested in answering. This configuration includes many things. It might include things like hyperparameters in the model or priors if you're in a Bayesian setting, but it also includes things like which variables do we include in the model and what transformations do we apply to the data before it goes in the model at all? One important thing, it's important to do this configuration before the model actually sees the data. We want to start with our assumptions and only then run the data through the model. The problem is, once you see the results, you might be incentivized to go back and tweak your assumptions to make the results look a little bit different. But this is bad science and it can lead to biased models that will lead you and your company astray. There's a reason that Bayesians call these kinds of hyperparameters priors because they're supposed to be the state of the world, your beliefs, the state of the model before it sees the data. Setting hyperparameters and priors for a complex media model is not easy. In general, if you're following best practices, you want priors that are going to be as uninformative as possible, meaning that they should rule out impossible combinations of the data or things that you know a priori cannot be true, but other than that should not impact the results too much. One important question is how do you even get the priors to begin with to elicit priors from the marketers that are setting up Recast? We have a set of questions that we ask that can get their beliefs about how marketing is performing and that can be translated into priors that we set into the model directly. This is how the model is customized for each customer and the data that they have. Here's Andy Bennington, head of strategic insights at Recast, walking through a few of these questions.

Every Recast model build process begins with a deep dive into the client's business. This deep dive ultimately boils down to just a few key questions. Now, before I get into those questions though, it's important to note that for all of them, we don't expect our clients to have the precise answers to them because if they did, they wouldn't need to be talking to Recast. Instead, we're asking questions so we can set those boundaries for the model that reflect the uncertainty that our clients have about their own business. All right, let's get into the questions. Question number one is how much of your business would remain if you turned off all of your marketing tomorrow? Now this is a very complicated question, but some of the factors we encourage clients to think about when answering are how strong is your brand compared to your competitors? Do you have an obvious distribution, advantage or disadvantage in your category? And just in general, how competitive is your industry? Remember, we're trying to cast a wide net here so that Recast can go and do the work from there. Question number two is how long does it take your customers to go through that decision journey when they're making a purchase? And which phase of the journey are each of your marketing channels allocated towards impacting? For example, some channels might be targeted towards earlier stages of that decision making process. So we want to tell the model to look for their impacts further away into the future from when you spent those dollars. The other end of that spectrum though is some channel might be targeted towards getting people right across the finish line. Maybe paid search is a good example here. And we want to tell the model to look for the impact within a few days of someone clicking on a paid search ad. It wouldn't make sense for the model to come back and say that paid search has its impact a year later. Question three is well outside of marketing, what other factors impact your business? This causing the broad, but we break it down into more digestible chunks like major promotions you've run in the past, some macroeconomic factors that might impact your business and business model changes or shifts to the way you've actually structured the way clients move through your website or your conversion flow. Those types of things wouldn't otherwise be captured within your marketing, but could have a large step change in the impact of your revenue. Ultimately, the goal of all these questions is to bring the client an actionable, rigorous model that can drive as much value as possible from day one all right, so those are the main three questions that we'll talk about when we're setting up a Recast model to make sure we're giving you a customized, actionable, and most importantly, a very rigorous model for you to drive as much value with Recast as possible from day one.

But if you're not using Recast, here are a few things that you want to make sure you don't do during this prior setting process. First of all, you don't want to just input data from from multitouch attribution or surveys or ad platforms. These methods are all biased and you don't want to make sure that their biases flow through into the model. Second, you don't want to just assume that what was true in the past is true today without verifying it first. We often see this as a source of confusion where marketing teams might not have thought enough about how things might have changed with respect to their performance since some exercise many years ago, and how things might be different today. That's very relevant to this modeling exercise. At Recast, we like and generally recommend setting broadly uninformative priors that don't bias the model too much. However, it's always important to write down the priors and to present the prior results along with the model results so that any biases in the priors are visible. We think that transparency is incredibly important and good modelers should be transparent about the assumptions being made, whether via priors, hyperparameters, or model structure. And so all of these things should be transparently available to anybody that is using the model. So in general, the broad principles for hyperparameter setting are 1. Start with the most important business questions to answer. 2. Then think about the relevant assumptions that you're willing or not to make within the relevant modeling framework. 3. Try to avoid biasing the model with priors or hyperparameters wherever possible. 4. Be transparent about the assumptions being made whenever communicating results. These are a good start to getting valid causal inferences out of your mediamix model. In the next episode, we'll talk about doing a configuration compatibility check as a way to make sure your assumptions are at least somewhat plausible.