Model Stability: The Most Underrated Quality in Media Mix Modeling (MMM)

Today I'm going to talk to you about something near and dear to my heart. And that is model stability. So at recast, one of our core philosophies is that MMMs should be used for forward looking decisions. That is, MMMs should be used to run forecasts, run optimizations, plan experiments, validate the results of those experiments, and so on. So imagine you're using a model that you update and every seven days to make these kinds of decisions. And one week it's telling you that Facebook is a really good channel and you should be scaling up into that channel. And then the next week it's telling you that Facebook is the worst channel that you've ever seen in your entire life and you should pull all of your money out of that channel. And then the next week it says Facebook is great again, and so on. Typically, if that was the MMM that you were using to make decisions, pretty soon you'd get frustrated, everyone else on your team would get frustrated and you'd put that model back in the shelf and you'd never use it again. Mmms or any model that produce really wild swings in output can't really be trusted because A it's indicating that there's probably something wrong with the model and B, from a practical perspective, it's just really hard to make long term decisions when over short periods of time the model is changing its output one week to the next. Now historically this was hard to see with MMMs because they would get updated maybe once a year or maybe once every six months, or if you were really advanced with it, once a quarter. With recast, we update our MMMs once every seven days. So every week we get a refreshed data set from our customers, we get seven new days of data and we trim the first seven days off and we refresh the entire model so we re estimate all of the parameters from the ground up with that sort of shifted by one week data set. So this is potentially a real problem for recast because we're re estimating the whole model every week. But we want to be aware of this idea that if we're telling you one thing one week, we should broadly be telling you the same thing the next week, unless we're of course learning some real signal. So with modern mmms, recast included, this idea of model stability is just a lot more important. Now, to show you why it's so important and to show you how it can go wrong, I've put together a simple little exercise with some code I've written up, I'll walk you through it just to show how model stability can go wrong when you're building statistical models to explain data. Let me walk you through it. The first thing that I'm going to admit is that I've chosen a seed for the random number generator that is chosen carefully to illustrate the phenomenon that I want to show. But you can try running it yourself with different seeds and you'll get broadly similar effects. So let's imagine that we have 1,000 time steps. That would be a couple of years of daily data. Now, the first thing that we're going to do is we're going to simulate our marketing spend. I've used a few tricks to make this a little bit more realistic. So, for example, this is the cumulative sum of a random normal variable with a relatively small standard error. What this is going to do is it is going to simulate a random walk. So what you obtain is a random walk. Now, here's the problem. We want to make this realistic. You can't spend negative numbers. So what this is going to do is it is going to bound the numbers between 0 and 1. And because the standard deviation is so small, it's actually randomly walking through a pretty tight range around 0.5. If I run this a couple more times, you'll see it's always going to be pretty close to 0.5. But you can see the idea here, which is that we're sort of evolving this over time, but in very small steps. And then I'm just scaling this by some reasonable amount to spend on a marketing channel every day. So I do this twice and you can see what the effects look like. Now I need to also do something with the roi. So right now all I've done is I've simulated our marketing spend. Right. So this is just the marketing spend over time. So we've scaled our spend into channel blue, and we've held our spend constant roughly into channel red. Maybe we tested them both out at the same time and pulled back on red and scaled blue. Might be a little narrative for what we see on the graph here for the ROIs. Again, we're doing almost exactly the same thing, except we are just using different values for our bounds. So this is imagining that the ROI of each channel has to be strictly between 0.01, almost no effect to 5x. Right. And so I'm going to run this and you obtain some plot of the ROIs, where, ooh, this is good. It works pretty well with our graphs here. This is of course random, but the marketer behind these Randomly generated plots. They had the right idea. They scaled into the blue channel because it was performing so much better. And they held constant on the red because it was falling behind. There was no actual explanation for what happened on these graphs. They're just randomly generated. This just happened to work well. And then of course, we simulate our dep var so we need some error. That's what this is. And then we have the pretty simple regression equation, right? So we have our predictors X1 and X2, and then our betas, beta 1 and beta 2, which here can be the ROIs of every channel. And then of course, some error term here. And so this would obtain some sales graph that looks like this, right? Some general patterns here, of course, some noise around those days. This is what we observe. And of course we also observe the spend. And then what we are trying to infer are these beta 1s and beta 2s. Now what I'm going to do is I'm going to show you how, if you are running a model and updating it every week, how it can go wrong and you can start to get, well, first of all, the wrong estimates of the betas. And then second of all, how those estimates can change pretty dramatically as you're updating them, even if the facts on the ground aren't changing that much. So what I'm saying is we're going to update the model every seven days. That's our offset, right? We're refreshing the model every seven days and we're going to repeat that refresh cycle 52 times, like in a year. So if we do that, the index of the first day that we will end the regression is on 636. And of course we're going to start on day one. And then these are just containers to contain our estimates of the intercept of the Beta 1s and the Beta 2s. And then in this for loop, and I can just sort of show you what's going on. Let's set I equals to one. This is saying, okay, actually let's set I equal to something interesting. I equals 20 or 32, right? So the 32nd time we update, the model will end at timestep 860 and we'll start at 225, right? There we go. And Y is going to be the part of the time series corresponding to those values, right? So if you squint between 2:25, we're zooming in on this block of time, right? And when we refresh the model, it will be at that block of time. And then when we refresh it again, it'll be at that block of time. So that's sort of how we're updating the model as we go forward. But the, what was it again? The 32nd time we started, we zoom in on the period of these black lines. Now you might be asking, well, why do you update the start time? And in this particular case there's no need. We're simply running a very simple linear regression. In recast case computational problems start to show up if you let the data grow. So we keep the window to a fixed size. So we now have our Y underscore, we have our X1 and X2 underscore which are the same idea. And we run a model and in this particular case we get our estimates and many eagle eyed participants might see something here which is that we have this really negative intercept estimate. Well, one big problem is that we know that the intercept here is zero. So we generated the data without an intercept and we're getting a very precise estimate of a very negative intercept. Intercepts shouldn't be negative. The default case of your business is not that you are giving money away, so it should be zero. And if it were non zero, it should be positive. Wrong on both counts in this particular case. But that's not why we're building this today. This is just as an aside, another way that these models can go wrong. Okay, so what we're going to do is we're going to run through this loop, right 50, I guess 53 times the way I've set it up. And then what we're going to do is we're going to store our estimates of beta 1 into this beta 1's vector and then we are going to show you what it looks like at the end. So let's run this all from the top and see what happens. Okay, I'm not sure where this plot, what this plot corresponds to must be this Y. So this is our very spiky series of sales over time. It'll look a lot less spiky if I put the Y limit to zero. So this is what it would look like if you set the axis to zero. So these are our sales over time, hovering around 25 to 30,000. But we've run this regression sort of 53 times and let's see what happens. So if you were at refresh 23, you'd say blue is better than red. If you were at refresh 24, you'd say, oh, now red is better than blue. And then if you were at refresh, call it 50, you'd say red is much better than blue, right? So over the course of this period, we have evolved our thinking from red is better than blue to blue is better than red back to red is better than blue. So over the course of this year, you have now changed your investment several times. Now you've gone from starting at red is better than blue to blue is better than red back to red is better than blue. Right. Now let's look at the actual betas. These are the things we're trying to recover. Now, note that they're changing over time, but in this particular case, remember that we start our analysis with this much data, right? So it's important to note that we start wrong. We start wrong, right? Blue is much better than red in the actual ROIs. This is what we're looking at right now. These are the beta ones, the actual ROIs. But when I look at the estimate of the ROIs, which is this, I get it wrong. I get it right and then I get it wrong. So only probably by happenstance am I getting it right in this period because the estimates are moving around quite a bit. In particular, the estimate of the blue channel goes from, you know, pretty good performance, almost a 4x ROI all the way down to actually below zero. Another problem with the model is that this is just ordinarily squares. There's no restriction on the coefficients. And so we can get this sort of nonsensical answer that for every dollar that you spend into the channel, you actually make less revenue. Not something that you should let your MMM tell you because that doesn't make much sense. But the more concerning thing is just that it is flip flopping over time. Or at least maybe it's not more concerning. But for the purposes of this exercise, this is the concerning part that I wanted to show you is that you can get this instability over time where the model is reaching one conclusion one week and another conclusion the next week, even though the actual signal is staying relatively constant over that time period. Again, remember the first time that the model updates, it is seeing all of the data from 0 to 600 and it's relatively stable during this period. And as it's updating right, blue is always better than red. However, while you update it, blue is always better than red during this period. But we get it wrong. We get it right and then we get it wrong again. We're making a couple reversals of what channel is better, but it's just, it's not getting the signal and it is flip flopping its recommendations. To you as the marketer. So I just want you to just imagine yourself, if you are a marketer, through the first, call it, 10 weeks of working with this model, you are feeling pretty good about it. You're getting consistent recommendations. You're seeing, you know, red decline in performance. Around week 10, you're saying, okay, maybe we've seen red decline in blue, you know, stay pretty constant, maybe increase its performance. This is good. But around week 20, you know, six months into the model, you just see a really sharp reversal in its recommendations and you're sort of wondering, why is this model flip flopping? Right? Can I trust what it's saying? Can I use these estimates? And in this particular case, you'd be absolutely right to question the model's recommendations because it has been giving you wrong recommendations for most of its useful life and will continue to give wrong recommendations if you were to continue to use this model. And beyond the frustrations, there's also real value getting left on the table. Here at Recast, we focus a lot on instability. Sometimes it does creep up in our models. Usually there is a good reason for the instability, meaning that there has been some really large signal that the model has picked up on, leading it to change its values. But what you've seen here is that sometimes a model can be unstable for no good reason whatsoever. It just means that it's not really picking up any signal. And so it is flip flopping between different random solutions. And that is a model that is actually worthless or actually worse than worthless. It's negative value because it is leading to a lot of frustration, it's leading to a lot of distrust, and the model results themselves are likely to be wrong. So that is why instability and model stability checks before we deploy a model are so crucial to RECASTS model validation process and why it's such an active area of research for us.