How to Use Lift Tests to Calibrate Your Media Mix Model (MMM)

You.

Hi, I'm Tom Vladek, co founder and co CEO of Recast, the world's most advanced media mix modeling platform. And it's where the world's best brands measure and optimize their marketing spend. Today we've got episode three of my Chalk talks. Today I'm going to talk about time varying covariates, parameter recovery. But the main topic I'm going to be covering is the impact of using lift tests to calibrate your mmm. Why they can be so powerful and beneficial. I'm going to walk you through a pretty in depth example of how it works on Recast platform if you follow my videos. A couple weeks ago I said that MMMs and lift tests go together like peanut butter and jelly are really complementary because MMMs can recommend which channels you should test the ones that are most important in terms of driving your business outcomes. Lift tests can also be used as information to the MMM itself and make the MMM overall more accurate even in channels that you don't test. I'm going to show you exactly how that works as I go through this example today. All right, I hope you're ready to dig into some code today because I'm going to go through it step by step, make it super explainable every moment along the way. I've got rstudio open here and I've got two files. One is a file that walks through all of the analysis and one that has a very simple model in Stan that illustrates a lot of the concepts I'll be discussing and is similar in concept to the MMM that Recast uses. Let's get started.

How do you.

I have some libraries I need. I'm going to be estimating things in command stand, I'm going to be manipulating data in dplyr and this last one, scoring rules. We're going to talk at the end why that's important, but this is going to be used to estimate the precision of the MMMs both with and without lift tests included. The first thing I'm going to do is simulate some data. This is super important. Always when you're testing model structure, the impacts of new things on your model, you always got to simulate data and see whether or not what you are going to include in the model, in this case lift test calibration is actually going to work. The only way that you know it works is with data that you've simulated because only in the case of data that you've simulated do you actually know the true parameters that drove the model. So I have this thing called a Kernel. What's a kernel? A kernel is a covariance matrix. In this case it's the Cholesky decomposed covariance matrix. But this shows what I'm doing here, and you don't need to know the specifics, is I'm generating data via a Gaussian process. I'm not going to get too deep into the weeds of how Gaussian processes work, but they're ways of generating realistic time series data. Our Gaussian process has is that today is like yesterday, which means the data we generate is going to be smooth, right? One day, following the next they're going to be close. So when you draw a line through all the points, it looks like a smoothly varying line. And the other assumption is that today is like a year ago. Meaning that if there is a seasonal pattern in the data, this Gaussian process should be able to fit it. The way to generate data from a kernel like this is just to matrix multiply it by a standard normal distribution. So the way to generate data from a Gaussian from the Cholesky decomposed kernel like this is just to multiply it by a standard normal distribution. And, and that will give you simulation draws from the Gaussian process. So that's what's happening here. I've got this kernel matrix, multiplying it by a matrix of normally distributed random variables. And then what comes out the other end are these draws from our time series, right? And you can see they follow like a smoothly varying pattern. So I'm going to use this general structure to generate time varying ROIs. And so what I've got here is a hard lower bound of zero, hard upper bound of five. I'm doing it for three channels. These control the mean and skew of the distribution because I'll explain what that means in a second of standard normal random variables. Here I'm going to generate some overall means and then what I'm going to do here is I'm going to take my time varying coefficients, scale them by some factor s, which makes them a little bit tighter. I'm going to add an overall mean to them and then a channel specific mean, and then push them through an inverse logit distribution to bound them by the caps. Once I've done all that, I get some ROIs that look like this. So this S is sort of dampening the movement. The mean is pushing them towards the bottom of the distribution so that they're a little bit closer to 0 than 5. And then this is bounding it within that range and I still get this smoothly varying over time pathway of the rois. These are some sort of realistic rois. I've generated this path of rois and I can plot them. They look pretty reasonable. The other thing is that I need to simulate spend. Here I take a random walk, the cumulative sum of normally distributed random variables calibrated to make it look reasonable. I can simulate spend that looks like this, right? And this actually works out to a, you know, fairly reasonable image of what spend might look like for a growing brand. And then to get the impacts at the channel level, I multiply the rois by the spend. I can plot these as well. I get one channel that's really dominating and two channels that are much lower and one channel that seems to be not performing very well at all. Finally, to get the outcome, I take the row sums of the impacts and then add some noise to it. I can get the outcome. This looks like a nicely growing brand, maybe with a little bit of a seasonal pattern. Overall, the shareholders of this business and people that work there would be very happy growing quite a bit in the past couple years.

Cool.

So now I've simulated data, so what do I have? I've spend and I have an outcome that's what we'll send into the model. And then I have these ROIS or these impacts that I care about estimating. Let's talk about how we do that in the model. This is Stan. For those of you that haven't played around with a platform like STAN or pymc, don't worry too much about it. I'm just going to walk through how this model is structured because it's basically structured the exact same way as I've generated the data, just with a few more pieces to it. So you've got to tell it the dimensions of the data, right? So I've got time, steps, channels, lift test will come to this. I'm passing in the same kernel that I use to generate the time varying data. I've got my outcome Y, I've got my predictors spend, which could be called X. I've got a prior on how much error there's going to be. I've got my priors on the ROIs, the upper bounds, the lower bounds, the means, the scales. I'm going to be passing through the lift tests. So this is going to tell you sort of which channel. When did the lift test start, when did it stop, what did we measure as the result during that lift test and what was the standard error of that estimate? Finally, I'm Going to be running this model, only generating from the prior to show you what the prior looks like. Meaning like if we don't actually estimate the model, we just test the model's assumptions. What does that imply? Very important to do when you're building a model this complex. Finally, what are we going to be estimating? We're going to be estimating the roi, the mean roi, the ROI over time, the impacts, the predictions, the error, and then I'm going to have a penalty for violating the lift test values. I'll walk through how this works. For every step in the model, we're going to be generating the ROIs. And how do we do that? Well, this line here should look pretty much exactly the same as this line here.

Right.

So we've got an inverse logit function, a lower bound and upper boundary. We're multiplying by a kernel. We have some overall mean within this inverse logit that we think is true for all channels. We've got a channel specific mean. And that should be exactly the same in here.

Right.

And so we're estimating what I'm calling ADA roi and eta ROI are just these latent parameters. And that's the same as here.

Right.

Every time I generate this random data, I can get a path of ROIs here. And, and so really when you sort of drill down, what you're trying to estimate are these ETAs. And that's what's here. If you can generate or estimate these, you can generate ROIs. If you can generate ROIs, you can generate impacts. If you can generate impacts, you can generate predictions. If you can generate predictions, you can get errors. And so that's the chain. Really the only thing that we're estimating here are the ROIs. And then that flows through all the way to these errors. The errors are what we're trying to minimize in a Bayesian way. But that's how you get the prediction errors in the model. In the model itself. What we're basically saying is we think that the errors are going to be normally distributed with some standard deviation sigma. We think that the sigma is going to be exponentially distributed by the reciprocal of the error scale that we pass in the ROI means. The sort of channel specific means are going to be standard normal, the ETA ROIs are going to be standard normal, and these are the things that generate the actual data, the ROIS that we care about. And then we'll talk about the lift test penalty in a minute. So this is how the model works. What's happening is that the model is Sampling values for these values for the ROI means, the ADA rois, the sigma, the errors, and then it's passing it through this set of functions to generate predictions. What it's going to try to do is it's going to try to find the ROI means and ETA ROIs that minimize the overall penalty on the model. If these errors are really wide, that's going to be a huge likelihood penalty. And so it's going to incentivize the model in a sense to find these ETA ROIs and these ROI means that do a better job of minimizing the errors, but that also correspond to the priors that we've specified. So what I'm going to do here is tell command stand to compile the model, I specify the data and this is going to be a prior only run with no lift tests. I just want to show you what this looks like. I could hit model sample and do all this. It does take a little bit of time. For the sake of efficiency, I've already estimated it. I've saved it in this RDS file. Luckily I already have it in memory, so I don't have to do anything. What I'm going to do is plot what the ROIS look like when I simulate from the prior. Takes a moment and you can see this is a bunch of draws from the prior. These are basically the same thing as if I just plotted a bunch of these when I did this. I plotted 10. If I plotted 100, it would look like that. If I plotted 1,000, it'll take even more time. Now the reason this looks slightly different than the image that I've showed you is because I haven't push this through the inverse logit function, added the mean, et cetera. But you get the same idea.

Right.

So what this is saying is that our assumptions before we look at the data are that the ROIs for channel one are going to look like this potentially.

Right.

So they're sort of weighted more towards the bottom end of the scale. They might go as high as 5, but we think that's unlikely. We can do this for channel two as well.

Right.

And it's going to look pretty much exactly the same.

Right.

This model doesn't know anything about channel one or channel two, how much you've spent, what the outcome looks like. It's just playing with the assumptions. I've now fit a model just with the prior only.

Right.

So prior only is one with the lift test included. Just to reset the stage, I'm plotting the ROI curves as they look with the no lift test version. And now I'm going to do the grand reveal. Without showing the model, the predictions of the error, anything like that. What do the lift tests do to the model when this plot renders? What you will see is a bow tie shape and the knots in the middle are where the lift tests are. We've gone from a world where all we know are our assumptions to now we've run two lift tests. We can see that during the period of the lift test, which I'm actually going to annotate here, I'm going to bracket them with vertical lines at the time steps where the lift test took place and horizontal lines showing a range of plausible values that the lift test came up with. We can see that the model. What we're in a sense doing is we're pinning the model to those lift test estimates. Now, between the lift tests, the model can be more flexible, but not infinitely. So I wanted to zoom in on the axes to show what that model learned. In addition to the lift test, before we included the lift tests, we have ROI values sort of above 3 and below 0.5 at the high end and the low end. But with the lift tests, even though we've only included across 810 days of ROI values, we've only included 60 days to 31 day lift tests. We now have essentially no values above 3 and essentially no values below 0.5. So we've gone from 0 to 4 to 0.5 to 3.

Right.

So we've actually learned quite a lot. Why have we learned quite a lot? Well, we have time varying ROIs which we think are very important because your channel performance can change over time. New competitors can come in, you can have raised your prices, you can have things like iOS 14 which dramatically change the advertising landscape, so on and so forth. So ROIs should be able to change over time, but they can't change in any way, shape or form. They have to change smoothly with a seasonal pattern. What that means is that even outside of the lift tests, we're still going to be learning information and that uncertainty is going to propagate as you go further away. And so even close to the lift tests, we're still similarly shaped to the lift tests or in a similar range. As we get further away from the lift tests, the uncertainty gets widest, but we still learn something about the overall ROI of the channel. We've learned that it really can't be above 3 and it really can't be below 0.5. In the region around the lift test, we've actually learned quite a bit more than that. But there are limitations. Like we really haven't learned anything with just the priority for the second channel. For example, the second channel is still just as uncertain as it was before. So now we're going to do the same thing. We're going to run two models, but we're going to actually include the prediction errors in the model.

Right.

So we're going to take off this prior only flag. What does that do? That shows the errors to the model. Right. So now the model is going to be incentivized to actually make its predictions close to the actual outcomes. So we run that and now what do we get? First I'm going to show you what that looks like. This is the prior with the lift test on the predictions it's going to render. And our outcome is going to be somewhere in there.

Right.

This isn't a very accurate model and for good reason. The model does not care about how accurate it is at this point. Now we are running a model where the model does care about how well it fits in sample and we'll show the predictions. The predictions are very tight around the actual outcomes, exactly what you would hope for. But let's take a look at the ROIs. So this is this model's prediction of Channel One's ROIs. They're pretty uncertain still. If you look at the range, we've collapsed the range quite a bit from 0.5 to 2.5, whereas before it had been 0 to 4. Right. So this is our state of the world before we introduced any information to the model. And we've gotten closer. If you look at the axes here, this is ranging from above four down almost to zero. And now we're in a world where we're sort of 0.5 to 2. So we've learned a lot about the channel, but there's still quite a bit of uncertainty. And we can plot what we've learned about these other two channels. This is channel two and channel three has the most spend, so it gets the most signal in the model.

Right.

So this is the most precise version of the model. And actually, let me just make sure. Nope. There we go. Yeah. So we have like almost 2x the spend in channel 3 relative to the other two. No mystery why we get the most Signal on Channel 3. But still quite some uncertainty here. Now, on the last day of the model, assuming that the ROI can be anywhere between 2.5 and 3.5.

Right.

I think a lot of people might consider that to be a very wide range. Similarly, with channel one, on the last day of the model, assuming you're at present day and you want to get an estimate of the ROI, you'd say it's between 0.5 and 2. The question is, how can we have this much uncertainty and still make predictions that are this accurate? Well, this is illustrating a number of things that are sort of bedeviling about MMMs. You can get very accurate or very tight predictions even with uncertainty in the channels that make up the prediction.

Right.

And the reason is that the uncertainties cancel out.

Right.

If you're overestimating high on one channel, you're underestimating low on another. It's quite common to have very precise predictions, but with uncertainty at the individual channel level. What this means is that there are a lot of possible versions of reality that are all consistent with what you ask actually observe. And what you actually observe is what you spend and what you get. There are a lot of different ways to go from what you spend to what you get. There's a lot of uncertainty in the channel's performance, and you're seeing that here.

Right.

We've learned a lot by adding lift tests, and we've learned a lot by adding the prediction errors to the model. Let's see what happens.