How to combine lift tests and experiments to improve your MMM | Modern Analytics Ep. 6

As marketers, incrementality is the thing that we really care about. Whether we're building a marketing mix model or running an experiment, what we care about really is measuring true incrementality. And since both marketing mix models and experiments should be designed to measure incrementality, shouldn't we be able to use them together to get even closer to the truth? I'm Michael Kaminski, econometrician, entrepreneur and marketing science researcher. Welcome to Modern analytics for marketers, our YouTube series that helps marketers like you learn and put into practice modern marketing measurement frameworks. This is the sixth episode in our series on how to validate marketing mix models. Today we're going to talk about how lift tests and incrementality experiments can be used to validate and improve marketing mix models. First, let's talk about experiments in the context of incrementality measurement and causal inference. A well run experiment can be a very powerful way to measure true incrementality, that is the true causal relationship between marketing activity and sales or whatever business KPI we're using. The reason why experiments are powerful is because they allow us to directly isolate the impact of changing one and only one marketing channel or strategy at a time. This direct manipulation of the marketing activity is very powerful and allows us to understand true causal relationships while only making minimal assumptions. This is in contrast with an observational method like marketing mix models where the levers we care about marketing activity aren't necessarily directly manipulated. For that reason, observational methods like marketing mixed models require more assumptions in order to make claims about true causality or incrementality. There are two main types of experiments that we use to estimate incrementality in marketing 1. Geographic matched market tests 2. Randomized control trials Both of these types of lift tests are very important and have different trade offs and limitations. In its simplest form, a geographic match market test works by dividing your addressable market into different geographic regions and then running ads for a given marketing channel only in one half or one part of those regions. Then you can compare how much lift in total sales you see in the regions where you ran the ADs versus the regions where you didn't. The relative difference in lift tells you about the true incrementality of running ads for that marketing channel. A randomized control trial works in marketing the same way it does in medicine. The idea is to take a sample of potential users or customers and then randomly split the group in two. You show ads to one group but not the other group, the control group, and then you compare the purchase rates between those two groups. If the groups were truly randomly split and are otherwise comparable, then any difference in purchase rates between the two groups can be attributed to the ads that the treatment group saw. That's the idea behind lift they measure incrementality with statistical uncertainty at a specific point in time, that is when you ran the experiment. Because these experiments isolate the effect of only one channel via experimental manipulation, they generally have less uncertainty, though not always on the read, of incrementality for that channel. This means that we can use the results of these experiments to help validate and improve the read that we're getting from our marketing mix model. But what does validation actually mean in this context? There are two ways that we think about using experimentation to validate marketing mixed model results. 1 Validating MMM results with an experiment after the MMM has already been fit. That is you get the results from your model and then you go and run an experiment in order to check to see how well the results from the MMM line up with those experimental results. 2. Using past historical results to improve the accuracy of the marketing mixed model via the model fitting procedure itself. That is, you use information from experiments in the past to help the model home in on the correct incrementality estimates that are consistent with the model, the historical observational data and the experimental results. Both methods are useful, but it's important to make sure you clarify which method of validation you're talking about when when you talk about using experimental lift tests in conjunction with marketing mixed models. Before we move on, it's important to talk about the limitations of using LIFT tests in experiments to validate MMMs. There are four main limitations you should be aware of. First, experiments are snapshots in time and may not apply to the entire time period you're analyzing with your MMM model. Second, some channels are impossible or very difficult to test with experiments. The third, experiments have uncertainty and unless they're perfectly run, they can actually lead you astray just as much as they guide you. And fourth, while experiments can be used to estimate average lift for a certain level of spend, it can be difficult to validate other parameters relative to the marketing mix model. So let's talk about the implications of each of those issues. First, experiments are snapshots in time, so you want to make sure that you're making the relevant comparison between your experiments and and your marketing mix model. If you run an experiment in the same marketing channel one year after another and you get different results, it's likely because that channel's performance has changed over that year. You want to make sure that when you're doing the validation, you're making an apples to apples comparison between what your MMM is estimating and what the experiment is reading. Second, some channels are difficult or impossible to test, so you simply won't be able to use this validation method for every marketing channel. A channel like Podcasts is famously difficult to test because there's no way to control which individuals or geographies get the ads or not. Similarly, local TV is bought differently than national tv, so it's actually quite difficult to design a test for television that generalizes well to a true national TV campaign. Third, it's important to keep in mind that while experiments are powerful methods for measuring incrementality, they're still subject to random variation, uncertainty, false positives, false negatives, et cetera. If the testing control groups aren't truly comparable, then you can end up getting totally false results from your experiment. So experiments are good, but just because the experimental results conflict with your MMM doesn't mean that you should throw out the marketing mix model without thinking twice or looking closely at why. And finally, there are other aspects of a channel's performance that aren't just lift. You may care about understanding the diminishing marginal returns curve for the channel, or the length of time between an exposure and a purchase within that channel. Or these are other aspects that can be more difficult to measure with an experiment. So it's important to make sure you know what you're validating and what you're not able to validate with an experiment that you're running. Okay, that was a lot of information on validating with tests. Let's sum up the most important lessons from today. 1. Both marketing mixed models and LIFT tests are designed to measure incrementality or causal relationships between marketing and your business KPI. 2 There are two primary different types of lift geographic match market tests and randomized controlled trials. 3. Lift tests are snapshots in time and include uncertainty, and it's important to take into account both factors when validating against your MMM. 4. Lift tests aren't a silver bullet, and so it's important to understand their limitations when developing a validation plan for your marketing mix model. If you found this video helpful or informative, please like it and drop us a note in the comments. It helps other people find this video as well. In our last and final episode on validating marketing mix models, we're going to talk about how to use prospective marketing budget changes to experimentally validate your mmm. Make sure you subscribe to our channel if you haven't already so that you don't miss it. In the meantime, happy modeling, and we'll see you in the next one. Sam.