All We See Are Shadows - Google Marketing Measurement Keynote - Michael Kaminsky, Recast Co-Founder

A few weeks back, I gave a keynote talk at Google about the current state of marketing measurement. The talk focuses on a hard truth. Incrementality is hard, and the truth of incrementality is unknown and unknowable. The only thing that different marketing measurement methods can provide us are shadows, glimpses of what true incrementality really is. I'll also talk about the correct way to interpret the results of these different attribution methods. And we'll talk a little bit at the very end about how to think about building an incrementality system that helps us not only learn as much as we can with the data that we have, but also take action to really drive more profit for your business. The end goal is to arm you with the knowledge and the frameworks that you need to improve your marketing program a little bit every single day. Hope you enjoy. All right, here we are. This is the talk called All We See Are Shadows. Very ominously named, but I am very excited to share this material with y'. All. Let's jump in. So to start marketing measurement, what are we even doing here? Why do we bother doing marketing measurement at all? The whole goal of marketing measurement is to drive more profit, right? And that's what I think we should really be keeping our eye on as we are thinking about different methods for doing mulcane measurement. The idea is that if we can measure the true incrementality of all of our different marketing channels, we can then design a theoretically optimal budget and actual I actually maximize the profit that we can generate for the business. So that's the goal. That's why we're here, is to maximize profit, given what we can learn about the true performance of all of our different marketing channels. So one important note before we jump in is that there are sort of two different terms that are often used to refer to the same idea. One is causal inference. That's a term that's often used by people with an economics background or econometricians, people who come from academia. They talk a lot about causal inference. In the sphere of marketing, we often refer to this same idea just as incrementality. Throughout this presentation, I will talk about causal inference. I'll talk about incrementality. I sort of will use those words interchangeably. Just so you know, they pretty much mean the same thing. So we're trying to measure marketing effectiveness. If we can measure all of our marketing channels, if we can learn about their true incrementality, then we will be able to make a perfectly optimal budget. Sounds easy. Enough, right? Unfortunately not. The most important thing to know about incrementality is that the truth is unknown and unknowable. We do not have access to true incrementality. We cannot access it here on Earth. And our attempts to measure true incrementality will always have trade offs. There will be sometimes things that work a little bit better, but it will come at the expense of something else. It is our job as marketing scientists to deal with this uncertainty and this just really difficult problem. That's what they pay us for. That's what we have to do. So here I want to introduce the fundamental metaphor that we're going to use throughout this presentation. Many of you have heard of Plato's allegory of the cave, and I think this allegory can help us to understand and appreciate our situation as marketing scientists and marketing analysts trying to learn about the world. We marketing scientists, us marketing analysts, unfortunately we are the ones that are chained to the rock here. In this metaphor, we can see evidence of true marketing performance, true incrementality, but they're just evidence. Those are the shadows cast on the wall, the true incrementality we can never observe ourselves. And unlike in Plato's allegory of the cave, we're stuck. We can't leave the cave. We have to do the best we can while only seeing shadows. So if all we see are shadows, then what makes a measurement method good? How should we think about evaluating different approaches to learning about the world and getting to this ideal of an optimal marketing budget? What actually matters is how close does some measurement method get us to a true understanding of reality, a true understanding of incrementality? That's what we should be focused on. That's what we should be thinking about, is we have measurement method A and measurement method B. What we should think about is which one gets us closer, given the context that we're in or the specific question that we need to answer. So there are three main measurement methods that modern marketers are using today. One, multi touch attribution, or as I like to call it, digital tracking. Two is lift tests or specific incrementality experiments, and the third is media mix modeling. All of these are aspirational approaches to incrementality. Sometimes, and I know some people in the marketing science world might say that this is heresy, Digital tracking or multi touch attribution is actually the best method for measuring incrementality. In other contexts, it's lift tests and experiments sometimes, very rarely, a media mixed model can help us understand true incrementality and when we're looking at all of these methods, we have to remember that all we see are shadows. None of these is the truth. They are all shadows. They are all approximations of the truth. So throughout this talk, I'm going to talk about some of these different methods. We're going to start with experimentation. So experimentation, is it the gold standard or is it just tarnished tooling? The dream, of course, is that controlled experiments provide clear reads on incrementality and causal relationships with minimal assumptions. The reality, of course, is less, less compelling. A couple of the problems that many of y' all will have encountered with running incrementality experiments or lift tests, that they're slow people tend to be bandwidth constrained with the amount of experiments that they can run throughout a year. There are often real constraints around which channels you can actually experiment in due to their addressability. They're noisy. This is a fundamental fact of running marketing experiments. Effect sizes are small, there's low signal. Often we're working with small insize. It means there's lots of noise, lots of uncertainty in the estimates. There's a temporal challenge. Experiments can miss out on marketing effects that are spread over longer periods of time. And of course, the results are ever changing. We know this theoretically, the macro economy changes, the marketing channels themselves change. And also empirically, results from experiments tend to change over time. So just because we ran some lift test experiment three months ago doesn't mean that those results hold true even today.

Right?

We need to constantly be experimenting in order to keep track of which channels are working for us and which ones aren't. So because of all of these issues, interpreting experimental results correctly is crucial. And unfortunately, I see a lot of problems with the way that many marketers try to interpret the results of these analyses today. So let's start with the wrong way to interpret a lift test.

Right?

In the wrong way of interpreting a lift test, we're focusing too much on the details of exactly what we're seeing, right? Where we're mistaking signal for noise, we're focusing on certain parts of the shadow instead of thinking about the true reality that is actually casting the shadow on the wall that we're observing. And we see this all the time. When a marketer says, hey, we ran a lift test, the lift was 4% and it's statistically significant. Unfortunately, that doesn't really mean anything. We need more information to be able to correctly use the results of this experiment. So the correct readout from an experiment is often something like the experiment resulted in an incremental cost per acquisition ranging from $50 to $1,200. And this sort of result is actually consistent with what we saw on the previous page, right? About 4% statistically significant, $50 to $1200. These are very different interpretations. And the business decision that we're going to make off of these results is going to be very different. So let's talk a little bit about experiments and their fundamental nature. And so what we need to take into account is we're both interpreting the results and communicating them. So the first thing to keep in mind about experiments is that they do not produce certainty.

Right?

4% lift, statistically significant, I think is a, it's a miscommunication because it makes it seem like it's the 4% that matters.

Right?

We are very confident the 4% lift is true. But that's not actually the right way to think about the results coming out of these experiments. Instead, we want to think about what are the range of parameters that are compatible with the data, given some uncertainty interval that we care about.

Right?

So what is the probability of the lift falling between these dots? What is the range of incremental return on investment estimates or incremental CPA estimates that are compatible with the data? Next thing to keep in mind is that experiment results show that performance changes over time. This is very commonly seen in empirical results and it should match our intuition as marketers that marketing channel performance changes over time. Right? So just because we run an experiment at one point in time doesn't mean that those results are going to hold forever. And perhaps implied in that, we have to remember that experiment results are time bound.

Right?

So we run an experiment, we get a read on during the time that the experiment was run. So April 1st to April 15th, we ran an experiment results. The results of that experiment apply most to the time when the experiment was actually run. April 1st, April 15th.

Right.

One day after, two days after, we probably have a fair amount of confidence that those Results still apply. April 16, April 17. By April 30, right. We should be thinking about, well, things might have changed a little bit in response to the experiment that we actually launched New Creative. And so maybe now the results are a little bit different. By the end of May, by June, right. We should probably be a lot less certain about how much we can rely on those experiment, on those experimental results that we ran all the way back in April. And so as we move both forward and backward in time in our own mental calculations for how we think about how trustworthy these results Are we should start to really plus place less and less weight on the results of that particular experiment. So does that mean that we should give up on experiments? We should not run experiments? Absolutely not. Experiments are good. Everyone that's watching this video should probably be running more experiments than you're running today. You should be thinking about how can you run more experiments. But critically important to keep in mind that they're not a magic bullet.

Right?

Experiments do not solve the causality problem. No method does. We're always working with shadows. We always have to understand the limitations of the measurement method that we're working with and the uncertainty that's implicit in this measurement method for accessing the unknown and unknowable true incrementality that we really care about. So what actually makes a good experiment?

Right?

Is it the confidence level, is it the power? Or is it the statistical significance? It's none of those things. I believe that experiments are good to the extent that they can provide support for, again, for or against certain hypotheses. So there's a context in which you could run a very underpowered, very noisy experiment, but the results of that experiment actually give you enough data to make some decision.

Right?

We're going to rule out pursuing this marketing channel. It might be a decent channel for us, but given our need to focus right now, we're only want to focus on it if it's an amazing channel. You can run a pretty rough and ready experiment that should give you enough signal to answer that question. And so in that case, a very underpowered experiment could still be enough. There are other times where you need a much more precise estimate in order to make some business decision. And then you need to design an experiment that can answer that particular decision. It's not a one size fits all question. When we're thinking about our experimentation program, we always need to be thinking about what is the business question we're trying to answer. What is the evidence that we need to rule in or rule out some state of the world and then designing experiments that help us get there as efficiently as possible. So what about media mix modeling? Magical incrementality machine or just sloppy snake oil? So the dream, of course, of a media mix model is we're going to use algorithms, AI to estimate the impact of all of our marketing channels. We're going to get true incrementality. It's going to be always on. It's going to just tell us what that incrementality is. We take data, we apply an algorithm, we get answers on the other Side, the reality is that media mix modeling is a magical, misleading machine. And the thing that all of you should take away from this is that it is almost always wrong. You should start from the assumption that your media mix model is wrong. Thing to know about media mix Modeling Training a media mix model is easy. It's always been easy. You can do it in Excel. It'll take you about 15 or 20 minutes. There's a ton of tutorials on YouTube. Go check them out. You can train it with one of the open source tools you can use Robin. You can use Lightweight, Meridian or pymc Marketing. There's a ton of these tools. They're just not that hard to get set up and run. Any of y' all can do it. There's a bunch of tutorials and walkthroughs that will show you how to train an mmm. The problem is that validating an MMM is the real challenge. All of those different methods for training in MMM will give you different results. With some of the open source tools, you just run the same algorithm on the same data twice and you get different results on the other side. How do you know which one's right? Are any of them right? That is the hard problem in media mix modeling and that's what you need to be thinking about if you're going to pursue this measurement method. So let's talk a little bit about why media mix modeling doesn't work as well as we might hope, right? There's a lot of other domains where we take data, we apply an algorithm, things just magically work on the other side. Why doesn't this. Why isn't this the case in media mix modeling? So there's a couple of core problems that I want to talk about. The first of them is overfitting, because media mix modeling is a causal inference problem, right? We care about estimating the value of some unknown parameters. All of the tools that you might be used to working with in other domains don't necessarily work. Additionally, marketing is a really complex field. There are lots of different ways that marketing has an impact on the world and can lead to purchases. And because of that complexity, we need to build fairly complex models in order to capture reality appropriately. And because we are working with very complex models, we have this problem of overfitting. And this is a really consistent problem in the field of mmm where we add on some new nice feature of a model that looks really good. Theoretically, when you write it down, it seems to match a lot of people's intuitions and observations in the world about how marketing actually performs. But the problem is that it generates overfitting. We fit to the noise and not to the signal. And so marketing mix. Modeling practitioners are constantly in this battle of trying to get enough specificity that captures the way the world really works, but avoiding the problem of overfitting, where the model fits to the noise and it can't actually make good recommendations or predictions about what's going to happen in the future. And of course, that's because it hasn't picked up true incrementality. Right? It hasn't gotten a read on the true causal relationships between marketing on one side and the business outcome that we care about on the other. Problem number two is model misspecification. And history nerds might sort of have some intuition about where this is going by seeing this image on the screen. This is a. A recreation of the Ptolemaic model of the universe. And Ptolemy was an ancient astronomer, I believe he was Greek, lived in Egypt, and he, like all of the leading scientists of his day, spent a lot of time thinking about, you know, how the universe is constructed. And he made the very obvious observation that he can plant his two feet on the ground, look up at the sky, and see the sun rotating around the Earth, see the moon rotating around the Earth, and even the stars in the heavens rotating around him from a fixed position. And so drew the very obvious conclusion that the world is the center of the universe and everything else rotates around the world. And he built some fairly complex models to be able to predict the position of the sun and the moon and the stars in the sky. That largely worked really well. You know, you can sort of build these concentric circles of the position of all of the different planets and the stars, and then use them to fairly accurately predict their position in the sky that you can observe just again, from standing here on Earth. The problem was that there were some times when these predictions didn't quite work. Many of you may have heard of things like Mercury going in retrograde, right? Which is where you from our position here on Earth, we can see the planet Mercury moving across the sky, and then it moves back and then it moves forward again. The problem was that the Ptolemaic model couldn't predict that accurate. And that was because there was a fundamental model misspecification in Ptolemy's model.

Right?

Ptolemy's model was never going to be able to predict that back and forth movement because at the very foundation of the model, it just didn't match the way the universe really works. And unfortunately, as media mix Modelers, we are in the same position as much as we can write down a bunch of pretty math that seems to be somewhat connected to the way that marketing has its impact on the real world, things like ad stock curves and saturation curves. We know that our models are going to be fundamentally misspecified, right? We cannot capture the true ways that marketing has its impact on the world. And so that often means that we are going to miss out on important features of marketing. And the problem with complex statistical models that we're using in the context of media mix modeling is that model misspecification doesn't just mean that you miss out on one part of the model. Oh, we didn't get the saturation curves right, but everything else is good, so we're okay. That's not actually how it works. If there's fundamental model misspecification, often everything is wrong. You don't capture saturation correctly, you don't capture the impact of promotions correctly, and every single parameter estimate in your model is going to be wrong. So that's a problem. The last problem that we face is the problem of unidentifiability. And some of y' all might recognize this illustration which Ludwig Wittgenstein used to help express this idea of unknowability about the world. And what you might observe is that if you look at this illustration one way you see a duck, and if you look at it the other way, facing the other direction, you see a rabbit. And the problem fundamentally here is that in this illustration, there isn't evidence to really say that it is one thing or it is the other. It is simultaneously both a duck and a rabbit. Because of the way this illustration is designed, we can't necessarily say it's definitely a duck or it's definitely a rabbit. It's both at the same time. And when we're building statistical models like media mixed models, we often face this problem where there simply isn't enough information between the model and the data to be able to rule one way or the other. Is it a duck or is it a rabbit? We can't rule one way or the other. Is television advertising better or is radio advertising better? Often this is the case in when we have lots of co linearity in our marketing program. If you always spend on TV at the same time as radio, there will be unidentifiability in the model. It'll be very difficult for any statistical model to be able to determine which one is truly better than the other. And so often we have this problem in media mixed Models where we just cannot answer certain questions with the data that we have and the statistical methods that we have. And either we need to just accept that and try to identify those cases where possible, or make some assumptions to try to rule in one state of the world or rule out another. Any of these problems, overfitting, model misspecification, unidentifiability, can cause the MMM results to be wrong. All of them. So the core challenges with media mix modeling is that, of course, the truth is unknown and unknowable. We don't know what the true incremental value of an additional thousand dollars invested in television is, which means that it's a causal inference problem. And you might be thinking to yourself, well, what about all of these measures of goodness of fit? Statistical significance? Still value of an additional thousand dollars invested in television is, which means that it's a causal inference problem. And you might be thinking to yourself, well, what about all of these measures of goodness of fit, statistical significance, R squared, Mape. Can they help me determine if my model is actually good? No, they cannot. Unfortunately, none of those metrics can actually help you determine if you are facing any of the problems that we just discussed. So in order to approach validation of a media mix model, we need both internal validity to be able to determine if this model could even possibly work at all. As well as measuring external validity, does the outcome of this model match other evidence that we have outside of the modeling framework itself? So let's talk about some of these checks. First, parameter recovery. This is the most important internal validity check that you can run. When we simulate data, when the true values are known, can our model get the answer right on the other side? Now, you might think this is really dumb. Of course, when we simulate data, we can also estimate the correct parameters. Actually a lot harder than you might imagine. It's especially for complex models. So this is a check that you absolutely need to be doing. Before you do anything else. Let's talk about what this looks like in practice. So parameter recovery, a couple of steps. First, we need to pick a realistic ROI for each channel. Again, this doesn't have to be related to what we actually believe the ROI is. This is a simulation exercise. The channel names don't matter at all. But what we want to do is we want to say, okay, let's simulate a world where TikTok has a 2.3x ROI and Facebook has a 7.7x ROI and TV has a 14x ROI or whatever, right? It doesn't matter. Just pick ROIs for all of these different channels that are different, and then follow the rest of the steps. Step two is we want to simulate our sales data or conversion data using that simulated ROI. So saying if it were true that TikTok had a 2.3x ROI and Facebook and a 7x ROI and TV had a 14x ROI, what would sales look like? And then we simulate those sales, and then now we have a data set right where we have our actual marketing activity. On TikTok, Facebook and TV, we have simulated sales data as if all of those ROIs that we chose were actually true. And then what we do is we feed that data into our media Mix model, ideally through the full modeling pipeline that we have, without telling the mediamix model about any of these about any of this simulation process. We run the model as if this world is true. And then what we do is we just check and we see if the mediamix model got the answer right. Did the model determine the correct roi, that is the parameter recovery check. And it's critically important to do. This can help us identify issues with identifiability, with internal consistency of the model, and most importantly, it can help us determine which assumptions are most important for the model. When you are doing this check, you should think about how can I break the model? Because models have simplifying assumptions, it should always be breakable. And so as you are doing your parameter recovery check, you need to be thinking about, okay, what are the combination of assumptions that would cause this model to break? Can we explore those? Can we understand how much seasonality matters? Check. And we see if the media Mix model got the answer right. Did the model determine the correct roi, that is the parameter recovery check. And it's critically important to do this can help us identify issues with identifiability, with internal consistency of the model. And most importantly, it can help us determine which assumptions are most important for the model. When you are those are great questions to go through as part of your parameter recovery exercise to help you understand which assumptions are most important. Now let's talk about external validity. So couple of different checks that we recommend that we think are really important. The first one is consistency with experiments. So we want to run other lift tests, other incrementality experiments outside of the modeling frame. Doing this check, you should think about how can I break the model? Because models have simplifying assumptions, it should always be breakable. And so as you are doing your parameter recovery check, you need to be thinking about, okay, what are the combination of assumptions that would cause this model to break? Can we explore those? Can we understand how much seasonality matters, market self and then compare those to the results that we get in the model. And then the second check is true forecast accuracy. Can the model predict the KPI that we're using as our dependent variable into the future on data that the model hasn't seen before?

Right.

These are the two most important external validity checks. So let's talk a little bit about how we compare MMM results with experiments. We can use the results from experiments both to validate the output of the model as well as calibrate the accuracy for future model runs. So if you're taking a Bayesian approach to your modeling, you get you end up with a really nice way of potentially incorporating experiments via priors. We think the right way to do this is using time varying parameters to estimate. Those are great questions to go through as part of your parameter recovery exercise to help you understand which assumptions are most important. Now let's talk about external validity. So a couple of different checks that we recommend that we think are really important. The first one is consistency with experiments. So we want to run other lift tests, other incrementality experiments outside of the modeling frame, how ROI changes over time, and then using experimental results as effectively data to constrain the prior only during the time that the experiment was run. So you start with uninformative priors for running the model. This should be best practice. And then you use lift test to constrain those priors during the time that the lift test was run. So forecast accuracy, we think this work itself and then compare those to the results that we get in the model. And then the second check is true forecast accuracy. Can the model predict the KPI that we're using as our dependent variable into the future on data that the model hasn't seen before?

Right.

These are the two most important external validity checks. So let's talk a little bit about how we compare MMM results with experiments. This is incredibly important. But sometimes people push back on this. They say media mix modeling is only about explaining why do we need to check forecast accuracy at all if all we're trying to do is explain? Couple of reasons. One is that you can't optimize last month's budget. The whole reason that we're doing this at all, going back to one of my very first slides, is that we are trying to understand incrementalities so that we can optimize. We can use the results from experiments both to validate the output of the model as well as calibrate the accuracy for future model runs. So if you're taking a Bayesian approach to use our marketing budget, if the model cannot predict the future, if it can't tell us what's going to happen next month, then it doesn't help us solve that problem. Second is we don't. We know that we or modeling you get, you end up with a really nice way of potentially incorporating experiments do not have the correct model. Our model is to some extent definitely misspecified. And so via priors we think the right way to do this is using time varying parameters to estimate. We have to check the forecast accuracy in order to validate if it's a usable model at all.

Right.

If we knew that we had the correct model, if we were working in a field like physics or chemistry, where there are fundamental laws of the universe that we're including our statistical model, then maybe you could say that explanation only is sufficient. But we're definitely not in that world. And so we need to be able to check out of sample predictive accuracy in order to believe the model at all. And then third is that our why changes over time and then using experimental results as effectively data to constrain the prior only during the time that the experiment was run. So you start with uninformative priors for running the model. This should be best practice. And then you use lift test to constrain those priors during the time that the lift test was run. So forecast accuracy. We think that if we've done causal inference correctly, if we're really estimating true incrementality, then what that means at a very deep philosophical level is that we should be able to manipulate inputs right our marketing budget and then be able to predict the outputs. That's what causal inference means. So if our media mix model is doing its job at all, we should be able to predict the future. So this is incredibly important. But sometimes people push back on this. They say media mix modeling is only about explaining why do we need to check forecast accuracy at all if all we're trying to do is explain? Couple of reasons. One is that you can't optimize last month's budget. The whole reason that we're doing this at all this is just resummarizing that last point. Causal inference means that we can predict what's going to happen when we manipulate an input. That is what we are doing in the context of a media mix model. So that is the test that we need to run in order to see if we've done this correctly. So what does this look like in practice, basically what we want to do is we want to split our data. All going back to one of my very first slides is that we are trying to understand incrementalities so that we can optimize up into some chunk of data that the model sees and then hold out some amount of data and validate if the model can predict the future on the data that it hasn't seen before. Some people are marketing budget. If the model cannot predict the future, if it can't tell us what's going to happen next month, then it doesn't help us solve that problem. Second is we don't, we know that we all think that looking at in sample model fit metrics like in sample R squared or in sample mape is good enough. It is not. We really need to validate that the model can predict data do not have the correct model. Our model is to some extent definitely misspecified and so that it hasn't seen before over some continuous time period. In order to validate the model, we have to check the forecast accuracy in order to validate if it's a usable model at all.

Right.

Has picked up true incrementality. So I've talked about a bunch of ways of validating media mixed models that often don't show up in the guides for how to run a media mix model. So why doesn't everyone do this? First is that not everyone knows about these techniques. Parameter recovery is really widely used in academia, but actually not that widely used in industry and by practitioners. If we knew that we had the correct model, if we were working in a field like physics or chemistry, where there are fundamental laws of, of the universe that we're including our statistical model, then maybe you could say that explanation only is sufficient. But we're definitely not in that world. And so we need to be able to check out. Historically, media mix modeling was just too slow to validate properly. And part of this was by design was that consultants would say, oh, it's going to take us three or six months to build a model and by the time the model has ample predictive accuracy in order to believe the model at all. And then third is that if we've done causal inference correctly, if we're really estimating true incrementality, then what that means at a very deep philosophical level is that we should be able to manipulate inputs right our marketing budget and then be able to predict the outputs. That's what causal inference means. So it's actually done. They're going to say, oh well, it's too late to validate because these results are now out of date. So there's nothing you can do. Sorry. And the third bullet point which aligns with the above is that a lot of consultants and sort of analytics insights providers have incentive not to validate the results. If you don't have a way to check the results, then you can never be wrong. And that, you know, provides you a lot of job security. It gives you a bunch of ways of amping up the amount of services that you're charging for. So that can be good for those consultants, not very good for businesses, of course. And the last is that it's really hard once you actually start running. If our media mix model is doing its job at all, we should be able to predict the future. So in these checks, you will see that most media mix models fail these checks. The vast majority of them, these checks are incredibly, incredibly hard if you do them correctly. So when does media mix modeling actually go right? When is it useful? Answer is almost never. Almost never it goes right. My belief is that media mix models should be assumed to be wrong until they are proven to be correct. Just because you run a media mix model doesn't mean the results are. This is just resummarizing that last point. Causal inference means that we can predict what's going to happen when we manipulate an input. That is what we are doing in the context of a media mix model. So that is the test that we need to run in order to see if we've done this correctly. So what does this look like in practice? Basically, what we want to do is we want to split our data right or even useful. You should start from the assumption that a mediamix model is going to be wrong until you prove that it is actually correct to the best of your ability. So if experimentation and media mix modeling are so difficult to get right, should we just give up? No. I believe that there's a better way forward. The better way forward is by taking a new approach to managing your marketing program. This is a thing that we've been talking about a little bit. It's through building what we call an incrementality system. So the idea behind an incrementality system is that we want to unite modeling and experimentation in this way. Observational models like a media mix model and experiment and an experimentation program can actually reinforce each other up into some chunk of data that the model sees and then hold out some amount of data and validate if the model can predict the future on the data that it hasn't seen. Before, some people think that looking at in sample model fit metrics like in sample R squared or in sample mape is good enough. It is not. We really need to validate that the model can predict data. So there's a couple of defining characteristics here. One is high velocity experimentation, experimenting really as much as we possibly can. A process of continuous hypothesis generate, generation and validation of model output. Dynamic marketing optimization. Continuously changing our marketing budget allocation based on what we're learning and then use that it hasn't seen before over some continuous time period in order to validate the model has picked up true incrementality. So I've talked about a bunch of ways of validating media mix models that often don't show up in the guides for how to run a mediamix model. So why using that optimization as a means of model validation, checking that the model is consistently. Doesn't everyone do this? First is that not everyone knows about these techniques. Parameter recovery is really getting it right when it comes to being able to predict the future as we are making these changes. This is a diagram explaining what this looks like in practice. We talk about these four stages as making up this cycle. The first is planning, right? Making a plan about how we're going to allocate our marketing budget. What experiments though were really widely used in academia, but actually not that widely used in industry and by practitioners. Historically, media mix modeling was just too slow to validate properly. And part of this was by design was that consultants would say, oh, it's going to take us three or six months to build a model and by the time the model is actually done they're going to say, oh well, it's too late to validate because these results are now out of date, so there's nothing you can do, Sorry, we're going to run. And that could be experiments in terms of just shifting marketing budget from one channel to another, or explicit incrementality tests, either GEO level or user level to try to get a measure on incrementality for parameters that we're uncertain about. Then we're going to validate, right? We're going to go run and we're going to go execute that plan and then we're going to see what we learned, right? Where did the media mix model line up correctly with what we expected? Where did it not? How are the experimental results reinforcing or not the results that we have from the media mix model? We're going to take all of that information, all those learnings, and use that to get more profit. We're going to optimize our marketing mix, right? We're going to move budget around and then we're going to go back to the top. We're going to make the plan for next quarter and continue. And the third bullet point which aligns with the above is that a lot of consultants and sort of analytics insights providers have incentive not to validate the results. And you don't have a way to check through this cycle of continuous exploration and exploitation based on what we learned in order to drive the most profit that we possibly can for the business. So marketing teams the results, then you can never be wrong. And that, you know, provides you a lot of job security. It gives you a bunch of ways of amping up the amount of services that you're charging for. So that can be good for those consultants. Future we'll focus on building incrementality systems, right? This means constructing a model of the world so that we can generate hypotheses, prove or disprove parts of the model, and then continuously test. The goal really is about generating teams that can generate and subsequently falsify hypotheses as quickly as possible. So that information from experiments not very good for businesses, of course. And the last is that it's really hard. Once you actually start running these checks, you will see that most media mix models fail these checks. The vast majority of them. These checks are can be used to continuously update marketing budgets and update the sort of core model of how the world really works. Which of our marketing channels are actually performing. So to summarize, we have to remember as we are thinking about our marketing measurement program, all we see are shadows. But we can learn a lot from shadows. They're incredibly, incredibly hard if you do them correctly. So when does media mix modeling actually go right? When is it useful? Answer is almost never. Almost never. We don't need the truth. We just need to improve our marketing program a little bit every day. Thank you. Hopefully this was helpful. Whenever it goes right, my belief is that media mix models should be assumed to be wrong until they are proven to be correct. Just because you run a media mix model doesn't mean the results are right or even useful. You should start from the assumption that a media mix model is going to be wrong until you prove that it is actually correct to the best of your ability. So if experimentation and media mix modeling are so difficult to get right, should we just give up? No. I believe that there's a better way forward. The better way forward is by taking a new approach to managing your marketing program. This is a thing that we've been talking about a little bit. It's through building what we call an incrementality system. So the idea behind an incrementality system is that we want to unite modeling and experimentation in this way. Observational models like a media mixed model and experiment and an experimentation program can actually reinforce each other. So there's a couple of defining characteristics here. One is high velocity experimentation, experimenting really as much as we possibly can. A process of continuous hypothesis generate, generation and validation of model output. Dynamic marketing optimization. Continuously changing our marketing budget allocation based on what we're learning and then using that optimization as a means of model validation, checking that the model is consistently getting it right. When it comes to being able to predict the future as we are making these changes, this is a diagram explaining what this looks like in practice. We talk about these four stages as making up this cycle. The first is planning, right? Making a plan about how we're going to allocate our marketing budget, what experiments the we're going to run. And that could be experiments in terms of just shifting marketing budget from one channel to another, or explicit incrementality tests, either geo level or user level to try to get a measure on incrementality for parameters that we're uncertain about. Then we're going to validate, right? We're going to go run and we're going to go execute that plan and then we're going to see what we learned, right? Where did the media mix model line up correctly with what we expected? Where did it not? How are the experimental results reinforcing or not the results that we have from the media mix model, we're going to take all of that information, all of those learnings and use that to get more profit. We're going to optimize our marketing mix, right? We're going to move budget around and then we're going to go back to the top. We're going to make the plan for next quarter and continue this cycle of continuous exploration and exploitation based on what we learned in order to drive the most profit that we possibly can for the business. So marketing teams of the future will focus on building incrementality systems, right? This means constructing a model of the world so that we can generate hypotheses, prove or disprove parts of the model and then continuously test. The goal really is about generating teams that can generate.