Robustness checks for your marketing mix model | Recast Office Hours #4

Too much variance that will cause lots of problems when you update the model regularly, as we do. And so you need to make sure that your model is robust with respect to the assumptions so that you're not getting a very particular set of results that are very contingent on just a few assumptions that you're making in the data, that if you could have made a slightly different choice, you would have gotten a very different set of results. And you also need to make sure that your model is navigating the bias variance trade off in such a way that when you get new data, you're not going to get completely different results. We'll talk about both here.

That is a great point, Tom.

Thank you. Cool.

So one of the ways that we do this at Recast is with this tool that we have built for doing what we call stability loops. And this is really specifically designed to test how do the model results change with relatively minor changes to the underlying data set. And so the thing, the basically the process here is pretty easy to reason about. You run the full MMM model building, model training pipeline a bunch of different times, slightly changing the subset of data that we're looking at. And so you can imagine running this in a loop where the first one you exclude, you run it on the full data set, the second one you exclude the last seven days, then you exclude the last 14 days, etc. You know, five, six, seven, 10 times, whatever number makes sense. And you can look at, okay, how do the results change as we are changing the data set in relatively small ways? This is actually really interesting because, you know, at Recast, we recommend retraining the model on a weekly basis. And a lot of other, like, people who do MMM are like, that's crazy. And they say that that's crazy for basically one of two different reasons that are sort of in conflict with each other. One of them is they're like, hey, you know, the results shouldn't actually change week over week, so why even bother updating it? And the other people say the results change way too much week over week, so you don't want to update it that frequently. And my point is sort of like, which is it? And if the results actually aren't changing very much week over week, then it shouldn't be a problem to refresh the model on a weekly cadence. What is actually the case is that a lot of times people are building these MMMs that are really not robust. Every time that they update the model with slight changes of data, they get totally different results. And that's a bad thing. Generally it means that the model is misspecified or isn't working in some fundamental way. And so this actually is a really important check to verify the robustness of the model results outside of just the minor subset of data that you might.

Cool.

So I went ahead and put it up. An example of this with lightweight MMM that we are going to take a look at. Actually, as I'm pulling up my screen here, are there. Are there people on the call who have done a bunch of work with lightweight mmm? I'm curious. I can't see the chat, so I don't know if people are chiming in.

Chat. Haven't gotten any volunteers yet.

No, no one. I could have used your help. I will warn y' all that this was my first time, like really trying to, like, run models with lightweight mmm, and I will say that I found it very confusing. So if you are a lightweight MMM expert and you disagree with some of how this is implemented, I probably agree with you, but it doesn't matter for the purposes of this example. So can y' all see my TMUX screen here with code on it?

Yes. No. Yes.

Okay, great. Thank you. So, talking to the code here, what does this actually look like in practice? And actually, just so that you all understand how my screen is laid out, the left hand side is code, top right is python, bottom right is just bash. So that's what you're looking at here. So, okay, let's take a look at the code. So most of this is just taken from the. The lightweight mmm like tutorial walkthrough. We import the libraries, we load in some data that we're going to use for the mmm. In this case, it just has a bunch of marketing spin data and then some KPI variable. These are the names of the marketing channels that we have. So linear TV mailers, meta prospecting, et cetera. And then we scale the data according, again, just according to the docs. And then we run the MMM model. Right. Relatively straightforward. Not a ton here we can see what the output of this looks like. Let's do mmm, print summary. It looks something like this, right? We've got all of these different parameters. The ones that I am really interested in for the purposes of this example are these, which is the coefficients on the media. We've got, you know, some channels that aren't so good, you know, and this is like a scaled roi. So I don't even know how to interpret this exactly, but lower is worse. So we have some channels that are bad and some channels that are Good. And we can sort of look at what that difference is. This is the output from this training. And then the question that we want to answer is, are those results robust? And so the way that we check this is really straightforward. It's just with a for loop. And so I updated, I basically created a second version of this script, does the same thing, but just puts all of the model training code in a loop. And so here we're looping through this trim days variable, so going from 0 to 36 by 7. So we'll get 0, 7, 14, 21, 28, 35 days back into the past. We deep copy the clean data and then we trim off the last N number of days. And so we're just going to rerun the exact same model, but with 7, 14, 21, 28 and 35 fewer rows in the data set each time, all of the other code basically looks the same, except then we take the MMM results and we append them to a list so that we can keep track of what the different results are. And I'm not going to run this code because it takes a couple of minutes again, because it's, you know, doing all of these different full MMM training runs. But the idea is that once we do this, we can see what the MM results are for the first run. And here I think I looked at this media channel one here is one of our best channels. But when we look at the fifth run, so with 35 fewer days of data, it's one of our worst channels. Those results are totally inconsistent. This is a relatively minor change in the total amount of data. And all of a sudden we get totally different results. This model is bad and cannot be used for decision making. I probably screwed up how I set up lightweight mmm. So, you know, I wouldn't necessarily take this as, like, ground truth. This is a common feature of most mmms that we see. They are not robust in this way. And so it is very, very important that, as practitioners, that you are running these checks to make sure that your results actually are robust. That when you do slight changes to the data, even at so much as adding one single day, one single row to the data frame, can actually have these really large impacts on the results and the implications of the results. And so it's very, very important that you're running these checks before you show these results to anyone or ask a marketing team to go make changes so that you know that the results are actually robust. And this example that I showed, get back to the code here. Right. Obviously only has a loop around, you know, the number of days that we're going to trim off the data set. But you could also imagine adding additional loops here to test the sensitivity to other assumptions. Right. Test the sensitivity to the assumptions of using the day of week seasonality test. Sensitivity to the ROI prior assumptions. Right. There's a lot of different versions of this that you can do. And building the framework for not just running anim, but rather running a bunch and seeing how sensitive are your results to these different assumptions is critically important for understanding how the model is actually working. And, you know, what are the key assumptions in this model? And if there's one really critical assumption that dramatically impacts the results, you want to know what that is. So that way you can either, you know, share that with the stakeholders or start to investigate, you know, what, what is the right, the real true right assumption that we want to make.

For instance. Cool.

So I showed you. I sort of flashed up some code on the screen so that you could take a look at it. It's obviously sort of very hacked together, but I wanted to do it in a simple way with an open source package so that y' all could see how this works. As I mentioned, we built some infrastructure for doing this at Recast that I think is really cool. So Tom is here in just a second going to share his screen to talk through the results that we have built for running these different stability loops and how we think about it.

So I'll pick up the baton. So, you know, as Michael said, he ran a loop of, I think, you know, call five models, compared them on one dimension, and found that, you know, one channel was the best, one week, the worst, you know, four weeks later. If you were using that with a real stakeholder, a real client, they would be asking you some tough questions about whether or not they should have been increasing spend in channel one four weeks ago based on what they know now. So we sort of, you know, as Michael mentioned, we take this very seriously. And this is one of the things that one of the checks that we do ahead of telling a customer that their model is at least past our checks and ready to be deployed. And so this is what we call a stability loop. Typically, we'll run it for six weeks. You're only looking at three right now. This is for demo data for, you know, a model that we've simulated with acquisition as a primary KPI. And what we're looking at here are overlaps, the average overlaps in the posterior from week to week. Right. And we've decomposed this from the sort of overall. Right. So on average, if one posterior is here, the next one is going to overlap it by about 90%. And we've decomposed it into what we call, you know, this is sort of old terminology, but the impact of their sort of main marketing channels, the impact of the intercept, the impact of holidays and promotions and the impact of lower funnel channels. And so on average it's 90%. It sort of decomposes this way. Lower funnel effect is obviously a very small portion. And then we look at it week by week. So the week ending in November 11, relative to the week before had like slightly less stability. The two subsequent weeks had much better stability. And what that looks like visually can be looked at in terms of their forecast and how they perform. Like what is the overlap of their forecasts or their predictions over that time period. We can also look at things like what is the aggregate effect of channels look like week over week? Right. We can see that there is a slight revision here. And we're sort of saying that in this case red is old and blue is new. That the later model revised the overall effect of marketing spend up and it revised the organic estimate down and had very little revisions to holidays and promotions. You might be wondering why we include total spend here. The reason is, is that we not only look at this sort of upfront when we have one data set and we're doing the model checks before we go to a customer and say, hey, your model is passing our checks. We do this every week as well. And it's not uncommon. In fact, it's more the rule than the exception that some of the spend data that they're passing to us will have changed in the meantime. And we're going to still look at robustness even when the data changes somewhat. So we're always removing seven days at the end and adding seven days at the beginning, removing seven days at the beginning and adding seven days at the end. But sometimes there are changes in the middle as well. So these are sort of the sort of very high level visuals that we look at for stability. But we also drill down to specific channel levels, which I'll come down to here. We have a summary here that shows which channels had less than 85% overlap of their posterior on average. And so there are times where linear TV or Facebook prospecting or podcast or Facebook retargeting revise. We can see that Facebook prospecting shows up in this sort of red flag zone twice over the stability loop. And so that means that we're going to drill a Little bit deeper into it. Every other channel that we're modeling apparently did not and had more stability. So that means that we go to a visual. We go to a visual. This is what we're looking at. This is simulated data. So there's probably a stronger weekly effect than what we would see in the wild here. So it's making it a little bit harder than normal to investigate. But in general, this is probably passing a smell test that even though it's throwing up a flag to investigate a little bit more thoroughly, that the overlap here in terms of the effect of Facebook spend is not, you know, giving us too much heartburn. And we think that this is okay, even though numerically it didn't pass the test. This is sort of like, okay, investigate further. We also have ways of looking at things like context effects, which are multiplicative effects, typically things like price or brand awareness, which are the two variables you can see here and how much they change week over week. Typically these variables are a little less stable, um, but we still like to investigate them and make sure that they're, you know, the overall learnings and effects that we're estimating from them stay relatively similar week over week. The reason that they're less stable is because they show up in the effects of the channels. And so as long as the effects of the channels are staying relatively constant, that the overall learnings of the model are, but the way that it's reaching conclusions might change a little bit more week over week. So these are kind of standard visuals that we might see for context effects. But the important thing here is that this is something that we do de rigueur before on every model, build every candidate model before it goes out to a customer to make sure that when we add seven days of data, we add 14 days of data, we had 21 days of data, that we're not going to get dramatically different estimates either on an overall level in terms of the effects of marketing, the effects of organic, or at a channel specific level, or some of the ways that we estimate those like context effects. And then every week when we update the model, we're going to look at this as well and make sure that even in the face of some data changes that we're still reaching roughly the same conclusions, that the model is not jerking the wheel too sharply. So with that, I'll hit pause. I know there's some comments. I don't know if there are any questions in the chat, but yeah, I can leave it there for now.

Thank you, Thomas. I don't think we have any questions yet in the chat. So which means that we have about 20 minutes for open Q and A. Tom and I are happy to talk more about robustness checks and how we think about doing this, but we're also happy to talk about anything broadly that could be helpful for y'. All. So any questions that y' all have, any thoughts, let us know. We're happy to help however we can. We need like the Jeopardy. Theme song going so Countdown here, maybe to spur it. So Bryce asked the question, could the changes in channel parameters be due to changes in spend rather than the model specification? Tom, I don't know if you have.

A. Yeah, so the answer is not really. So the reason is that we estimate a effect curve for any given level of spend for every channel for every day. And so as you move along that curve, you're going to get a different roi. As a rule, as you increase spend, your ROI will decline because your marginal effect of additional spend is decreasing. So to the extent that one day you're spending 100 and the next day you're spending 105, we're going to expect your ROI to go down even if the parameters stay the same.

And that's true in like all of the open source MMM packages too. Right. They're estimating a saturation curve. And so the channel effectiveness shouldn't change just due to spend level changes.

Yeah. Now an exception to that, Bryce, is to the extent that, you know, let's say we update a model on September 18th and we updated a model a week ago on September 11th, but in between updating the model on September 11th and September 18th, they updated the podcast spend in July. Right. And so the actual, the data, the overlapping data that we sent him to the two months changed. Now that of course is going to change the parameter estimates of the model, not due to the new data that we added, but due to the old data. Well, in, due to both. And so that could affect the parameter estimates. As a general rule, we're sort of less worried about that instability because it's sort of out of our control. We're really only worried about like as you add new spend. But practically speaking, since that is such a common thing that we experience, we do look at that anyway just to make sure that like, oh, if you change spend by sort of less than 1000 bucks overall, you shouldn't be getting dramatically different results. But as a general rule, changes in spend shouldn't impact the parameter estimates too dramatically. It should impact higher level results like roi, not the underlying parameters, but the exception is, of course, if they change past spend, if they're sort of sending you a different data set than they sent you the week before, I guess.

So we have some more questions coming in the chat, which is great. Keep them coming. I would love to hear. Do y' all do, do the people who are practitioners on the call, do you do any sort of robustness check for your model in your model as part of your, your model building process? And then Tom Adnan has this question about does the model fit also change and needs to be checked weekly, which I think we could maybe interpret a couple of different ways, but I'll let you take a stab at answering that how you want.

Yeah. So to this question, we at least update the model top to bottom every week, right? So we typically are estimating on about 810 days of data. So two years plus 60 or two years plus 90. And so we're going to send a full complement of data to the model this week and it, it's going to overlap in all but seven days with the data that we sent last week. But other than that, the models don't know about each other. Right. They are just totally new estimates from the ground. And yeah, so we need to check to make sure that the model is accurately retrodicting past data. So it's fitting the in sample data. We also check to make sure that it's predicting well. So we do a holdout check every week and so on. To your point, I mean, at least we, I know that not every MMM vendor does this, but we are fully updating the model every week and that means that we do need to check to make sure that it is still fitting the data well, that it's still predicting out of sample data well every week, in addition to this topic of discussion today, which is making sure that the internals of the model match the internals of the model last week, which is this concept of stability and robustness.

And so Marcelo asked about adding noise onto the data on these robustness checks. And I think that what you're getting, well, there's a couple of different ways of thinking about doing this. So in general, we're not going to add noise to the data other than, again, to the extent that you might think about randomly permuting, like what subset of the data the model is being trained on is sort of in some sense adding noise. You could think about adding noise by like shifting the priors, for example. I think that makes a lot of sense.

Right.

Like you sort of randomly tweak the priors small amounts for different key variables and see how that impacts the results. Adding noise to like the KPI variable doesn't seem exactly right to me, but I can't put my finger on why necessarily. Maybe that would be okay. I'd be curious to get Tom's thoughts.

You could do that. You would expect to get the same underlying parameter results with maybe slightly lower effective media, slightly higher effective intercept, and slightly higher error term. That would be my guess as to what the results would look like if you added noise. That, that could be a good idea. We, we don't do it because the way that we check for robustness is the way that the model like typically we're going to get restatements of spend, right? If anything, we're not going to get restatements of the dependent variable unless they're shifting to a totally new KPI, in which case you have to sort of rebuild the model anyway. But the like use case for us. The, the thing that maps to the actual problem is not, oh, the KPI is changing a little bit. Are we going to get more of the same results? It's going to be, oh, we're dropping the first seven days of data and we're adding the next seven days randomly.

Permuting like what subset of the data in that way.

So that's the way that we sort of shift up what data we're showing the model and making sure that we're getting broadly similar results. But that's not a bad idea, Marcel. I mean, we could do that.

It would test like all amounts for different key variables and see how that impacts the results. Adding noise to like the KPI variable doesn't seem exactly right to me. But I can't put my finger on why necessarily maybe that would be.

And that way of doing it doesn't map to a real world use case to us. It doesn't mean that it doesn't make sense. I think it does make sense. Underlying parameter results with maybe slightly lower effective media, slightly higher effective intercept, and slightly higher error term. That would be my guess as to what the results would look like if you added noise to the dependent variable could be a good idea. We, we don't do it because the way that we check for robustness is the way that the model like typically we're going to get restatements of spend, right? If anything, we're not going to get restatements of the dependent variable unless they're shifting to a totally new KPI, in which case you have to sort of rebuild the model anyway. But the like use case for us, the thing that maps to the actual problem is not, oh, the KPI is changing a little bit. Are we going to get more of the same results? It's going to be, oh, we're dropping the first seven days of data and we're adding the next seven days of data. Are the results going to be broadly similar in that way? So that's the way that we sort of shift up what data we're showing the model and making sure that we're getting broadly similar results. But that's not a bad idea, Marcel. I mean, we could do that. It would test like, hey, is the model navigating the bias variance trade off in a reasonable way? The only reason we don't do that is just that it doesn't really map. We do it in a different way that maps to an actual real world use case for us and that way of doing it doesn't map to a real world use case to us. It doesn't mean that it doesn't make sense. I think it does make sense and it's something we could consider. But I just, we're already doing these stability loops and that I think is the core use case that we need to solve for.

Yeah, but I think broadly, if you're a practitioner, like that's a good thing to would be like adding some random noise to the dependent variable and then seeing how that shifts the results, especially if the noise is mean zero. Like you would sort of hope that it doesn't impact it much. But again, based on my experience, especially with some of the open source tools, I would expect it will change it a lot. So it's a good, it's a good check to do any other questions from y'?

All?

All right, well, if no one has any extra questions, we can wrap up a little early. As always, Tom and I are always happy to, to be helpful if we can. Send us an email, send us a note, come to the next office hours, send questions along if there's things you would like for us to cover and otherwise hope you all have a great week. Really appreciate the time, really good conversation. Hopefully you learned.