Validating MMM in a privacy-focused era | Jim Gianoglio (Cauzle Analytics) | Coffee Breaks

Foreign.

Oh, there it goes. All right, we're going to jump in. We've got 30 minutes this morning. Want to keep it super efficient, keep the content really tight. We've got a lot of people here, which is really exciting. Plan is that I'm going to do quick introductions. Jim and I are going to jump into content. We've prepared some slides that we think are, you know, cover the most important parts of this idea of validating an mmm and if you have questions, drop them into the zoom chat. Marty on our team is going to be collecting those. We will have about 10 minutes at the end to try to answer questions that we see other questions that we don't get answered today. We will, you know, Jim and I will be working on content to address those in the future. So that will, you know, we'll, we'll make sure that we get to everything eventually. Also, this call is being recorded so we're going to share back out the recording of this. If you have other people who you want to share it with who couldn't make it today. Um, and I think that's it. So jumping right in with me is my good friend Jim Ginoglio. Did I get that right?

Jim, you nailed it. That's great.

I was practicing that pronunciation in the morning a couple of times today. I'm glad we got it right. Jim is a marketing mixed modeling expert. He has been around doing this for a very long time. He runs a newsletter called MMM Hub. We'll share out links to both recast stuff in a follow up email. He's an expert on mmm. He's also just started his own consulting firm called Causal Analytics. We are super lucky to have Jim with us today and with that we're going to jump right into the content. Jim, you ready to go?

Yeah, let's do this.

Let's do it. All right, validating an mmm. Let's get started.

Okay, so I'm sure all of you have heard of MMM before. Just by nature of being on this call, you've, you've seen those three letters together. But let's just give a really quick recap of what we're talking about today. So with marketing mix modeling or media mix modeling and you'll sometimes hear it referred to as econometrics. Basically what we're talking about is just applied statistics for estimating marketing performance, right? So we're looking for patterns in the data. Like if you increase your spend in Google search one week and you notice that your sales increase and then the next week you decrease your spend and your sales also decrease, then there seems to be a relationship there. Right. And so that's what MMM is trying to capture. Independent variables, you know, the, the spend in various channels to the dependent variable, which would be sales or revenue or leads, whatever it might be.

Right.

And this is not a new technology or methodology. It's been around for a very long time, since the 1950s. Typically, you know, early on it was the realm of large CPG companies because they had a ton of money and, and data scientists, or at that time they were called statisticians that could do this. But you know, one of the reasons we're seeing a resurgence in recent years of MMM is due to the changing privacy landscape and technology changes that come along with that, notably from Apple, with all of their fun things like ITP and ATT and with iOS 14 and 14.5 coming out and all the changes there that relate to data that's sort of being taken away or harder to get for us. So with mmm you just need aggregate data.

Right.

What does that mean? Well, you don't have to track at the user level, like with multi touch attribution. That can be very difficult given those changes that Apple has made recently where, you know, historically multi touch attribution, you would need to say this person, this human being, this actual user interacted with our display ad. They saw our display ad over here on this site and then they clicked on our pa, our Facebook ad, then maybe they clicked on our email link and then they clicked on a paid search ad and then they converted and being able to connect all of those touch points back to that one person. That's what multi touch attribution requires. But fortunately with mmm, we don't need that. We don't need that user level tracking. We just need impression or spin data from your channels, usually at a daily or weekly level, along with sales data at a daily or weekly level.

Right.

And that's, that's basically an oversimplification of the data that you need. But suffice it to say it's future proof from privacy regulations or technology changes. And also, let's not forget, we're not just talking about digital channels here, which traditionally with multi touch attribution, we're dealing with digital channels. MMM includes offline channels like out of home, direct mail, linear TV broadcast, all those quote unquote untrackable channels. So you can answer questions like how do billboards impact my sales at Walmart? And there are a lot of questions we can answer with mmm, but at the end of the day the main goal is to calculate the incremental return of every marketing investment. And that is how many sales did each channel drive that we wouldn't have otherwise gotten.

Right.

And we can use this data to help us better plan our marketing investments in the future so we can optimize our budgets to achieve the greatest return. It's about getting that right mix of our channels, right? The one, the, the mix of budgets across our channels that's going to give us the best return for our money. Now we get into the fun stuff. What makes MMM so special and so, you know, difficult from a validation perspective. And there are a few things that make these models harder to validate. For starters, we don't just want to know what variables are correlated with sales, although that's a good place to start. We actually want to understand the causal relationship between the two. And causal inference is a much harder problem to solve. Still a very active area of research. MMM uses statistical methods that are non deterministic, meaning you can get different results given the same inputs. And the actual truth, Capital T. Truth is not known. And this may sound disappointing, but it's generally unknowable. We can reduce our uncertainty in how our marketing is performing, but it comes at a cost. We need increased data collection or increased complexity in the model, et cetera. And it's a complex system with many variables that are beyond our control.

Right.

What are our competitors are doing? What are the general mindsets of our customers? Like, what are they thinking of day to day? Like, there's so many things that can change how effective our marketing is that are completely out of our control. And that truth is always evolving. And because of that complexity, there are infinitely many models that fit the data equally well, but they can give different conclusions. Right. So you might have one model that, you know, has, you know, accuracy metrics that are really good, and it says increase spend in Facebook and decrease spend and paid search. And you have another model that says, no, no, you should increase in paid search and decrease in Facebook. It says the exact opposite thing. And both of those models can fit the data equally well. So how are we supposed to know which model is, you know, quote, unquote?

Right.

Well, before we dive into what does work to validate your mmm, let's talk a little bit about what doesn't work and hopefully dispel any preconceived notions out there. First of all, we're not relying on checks of statistical significance, although that can be useful when you're running a hypothesis test, for example, comparing the difference in means between two groups. That's really not what we're doing with mmm. And another big mistake people make, and I see this very frequently when they're getting started with mmm, is they're looking at in sample goodness of fit metrics like R squared or mean absolute percentage.

Error, or.

Root mean squared error, normalized root mean squared error, et cetera. You never want to use these measures when they're based on the same data the model was trained on.

Right?

And some of them are easy to manipulate, like R squared, for example. It's relatively easy to manipulate a model to have an extremely high R squared just by adding additional variables to the model. Sometimes you'll see this happening where people just continue to add some dummy variables just to increase the quote unquote accuracy of the model. And lastly, cross validation, like leave one out or K fold cross validation. These are typically used with many machine learning methods. However, we can't easily do that with MMM because we're dealing with time series data, which just makes everything so much more fun. So that violates the assumption of IID or independent and identically distributed data. So it makes, you know, the sort of easy forms of cross validation really hard to do. So there are other ways to do it, but generally kind of naive. Cross validation is off the books here. And most importantly, none of these techniques actually differentiate between correlation and causation, which is what we truly want to understand. So how should we validate our model by, I would say by using one of these three methods here. One or more ideally, you could use all three of these methods to really truly understand and how your model is performing. So the first one is using experiments, AKA conversion lift studies, geolift experiments, holdout tests, randomized controlled trials. I'm probably leaving out a few other names that these go by, but this is our ground truth, right? This is the gold standard for understanding causality and we can use this to calibrate our models. Second up is out of sample or holdout forecasting, which is where we use the model to make predictions about data the model hasn't yet seen. And then we measure the accuracy. And the third one is dynamic budget optimization and evaluation, which involves changing your budget and evaluating the model's ability to predict the results. Now, that's all I'm going to say about this. I'm going to turn it over to Michael now to go a little deeper on each of these three methods to get you headed in the right direction.

Michael, that was great, Jim. So let's talk about these three methods and how we think about them and how we use them and how we can use them practically. So the first is validating with experimental data. We believe that the MMM should be set up to measure incremental mentality, which means that it should be validated by these other tests that you run outside of the model. And there's different ways of thinking about incorporating this information. So if you're using like sort of a traditional or a legacy approach to MMM, maybe you have 10 or 20 different models that you know, all have good fit metrics, yield different results, and then you winnow down those that set of models to the ones that are consistent with the the experiments that you've run. Or if you're using a Bayesian framework, right, you can actually include the information from those tests in the priors in your model. And there's other ways that you can also incorporate this information in other frameworks for either selecting models or actually using the information from the test itself to calibrate the model and push the model towards, or push the modeling framework towards the set of parameter estimates or the set of models that are actually consistent with these other sources of ground truth that you have. I'll just make the point here. In my opinion, when an MMM and a well run experiment disagree, the well run part here is actually very important. So keep that in mind. It's normally the MMM that's wrong. The MMM is making a lot more assumptions than, than what the experimental than what the experiment is making. And so when you have experimental data that conflicts with your mmm, I generally say you want to go take a look at what the MMM is saying and why it might be wrong and why am I can conflict with a really good experiment that we have a lot of confidence in, which as Jim mentioned, is really the gold standard for understanding causality. So next method out of sample holdout forecasting. This is a thing that we take really seriously at recast and put a lot of thought into. The idea is to test the model's predictions on data that it hasn't seen before. So what we're going to do is we're going to sort of chop off some amount of our data. We're going to train the model only on, let's say up to three months ago. We've seen the last three months, but the model hasn't seen the last three months. And then we're going to ask that model to make a forecast about what's going to happen over the next three months and we're going to compare it to actuals and what we're going to want to see is that the model can actually predict out of sample on data that it's never seen before. This is similar to the idea, this can be similar to the idea of cross validation and these other prediction check techniques. But we need to make sure that we're actually doing holdouts, you know, at the end of the data because of the time series component that Jim mentioned. If you just drop out one random week in the middle of your data set, that's not a good enough check because the model has a bunch of other information about what happened before and after. And so we need this idea of holdout forecast accuracy in order to get confidence that the model is not over fit to the data that it has already seen before. And additionally, what we really want to do is be able to do this consistently, right? If you only do it one time, you can actually overfit to that holdout data that you, that you held out one time and you can sort of run the model a bunch of different times and then try to get to the point where, okay, this does predict out of sample on those that single three months. But actually an even better way to do that is to do this consistently over and over and over again. And in particular, if you can do it consistently over and over and over again in the face of interventions, right when you are actually making changes to your marketing budget, that's going to give you really good evidence that you've picked up true causal signal. So if we predict well in May and then in June, we pull money out of TV and put more money into Facebook and then we predict again, that gives us a lot of confidence that we haven't just picked up the correlation. We've actually started to understand some of the true causal structure in the data. That's really important for being able to make recommendations into the future. And then last, and this sort of rhymes with the other stuff that we've been talking about is we can, we should actually think about making those changes proactively, right? If we get results from the MMM that say, hey, TV's our best channel and Facebook is our worst channel, we should be able to next month try it out, right? Let's put some more money into TV and take some more money out of Facebook and see if the model continues to predict accurately and see if our overall marketing efficiency actually improves groups. The idea of dynamically using the MMM to validate the MMM is a really powerful idea and it takes us out of this world where MMM is a one shot analysis and it gives us Some results. And we have to sort of take that on faith and then, you know, maybe work that into our marketing strategy to this world where we can actually use it dynamically. More like you use the tools in the marketing platform or in Google Analytics in order to dynamically drive marketing improvements for your business. And so this idea, hey, let's take these results and then let's go and test them and then update the model and see what happens. And the more that we can do that, the more confidence it should give us that again, our model has picked up this true underlying causal structure and also it's simultaneously making our business better. And so we highly recommend that you get out of the framework of like, hey, this is a one off report and we're going to like think about it and maybe incorporate it, you know, show it to the board and then never actually use it to this world where let's actually use this in an interactive, dynamic manner and put the results to the test in the real world. Cool. We are at 9:19 my time, which means that we are just in time to get to the next section of this, which should be questions. So let's switch over hopefully. I have some questions here. Great.

We do.

Okay, Jim, you can help me with this one. What is the best way to choose the best controls when we have a list of exogenous variables apart from market knowledge? I'll let you take a first stab at that. I'm not sure that we're going to agree on this. I'll let you take a first stab and then I'll chime in after.

No, it's more fun if we don't agree that's the better option. So I was thinking about this. I think the way that I think about this is that I'm going to make an analogy back to my days of when I used to implement tracking, Google Analytics tracking on websites. And there used to be this sort of dichotomy between people who would say, just track everything you can possibly track every single click and scroll our user can do and just dump that data into a data lake somewhere and we'll use it eventually. And then there were the other side of people that would say, let's be mindful about what we track. Let's come up with a strategy about what are the important things to track and what that means to our business and let's just track those things. And I kind of feel the same way about this. I fall on the side of let's be mindful about what we track as opposed to we have this big, long list of. Of all these external variables. We have, you know, sporting event dates, and we have weather, and we have demographics, and we have all this data. Let's go ahead and just throw it in there and see what sticks. I'm. I'm not in favor of that. I'm more in favor of taking, you know, your team's internal domain knowledge about what would affect sales and trying to get that data. I don't know. Michael, what do you think about that?

Yeah, so I think, you know, I love some of the things that you mentioned, for sure. Like parsimony, I think, is a really good idea. I think a lot of modelers get into trouble by doing, like, the kitchen sink regression that causes so many problems for a bunch of different reasons that are really well studied in the literature. Right. Like, if you think about the replication crisis that's going on right now, I think some of the stats nerds are probably nodding their heads, and some of the other people maybe are less familiar with that. But a lot of those problems come from people not thinking hard about, hey, what are the actual relationships here? And what is the implication of dumping a bunch of variables into this model? If you have a ton of variables to choose from and you're just dumping them into the model sort of randomly and then choosing the results that look right, you're really biasing the model. That's not good statistical practice. At recast, we are really into causal inference and really into Bayesian statistics. And so we always start from a causal diagram. Effectively, we try to map out, hey, theoretically, what are all of the different things that are impacting sales in this model, and what question are we trying to answer? And we use that to inform exactly how the model is specified, because we don't want to accidentally control for something that changes the interpretation of our other marketing variables. And so you actually like just the kitchen sink approach of throw in all the possible variables that could impact sales is generally a bad approach and will give you bad results. You really want to think about, okay, what is the causal relationship here? What are the. Like, how do we interpret the results of a regression analysis and then use that to structure. What exactly question are we trying to answer and how do we structure the model to get there? Cool.

I think we mostly agreed. I think we mostly agree.

I think we mostly agreed. I put a little bit more emphasis on the. On the causal diagramming, which, again, we're, like, really into. But I think we're very much on the same page there. Okay. Another question. When using test Results to enhance Mmm. What does the input data need? Is it lift by week from the test? Is it using the lift from the test to adjust MM inputs from the marketing channels tested? I'll talk a little bit about how we do this at Recast and then maybe Jim can talk about how you've done it in other frameworks. So with the Recast framework, we estimate a time series model, which means that we estimate how marketing performance changes over time and then we integrate test results via Bayesian priors on those time series coefficients. And so for Recast, what we need effectively is like the channel start date of the test, end date of the test, and then the results of the test, which generally are like a point estimate and then some standard deviation or standard error around that. And then we use that information to inform the prior, but only in the time period that the test was run. So it could be that you run a test in January and you know, it looks great and then you run a test in June and it looks bad. And basically what Recast will do is it will say, okay, we know these two points and then we can interpolate between them. So there must have been some evolution where this channel got worse over that time period. We think that's a really nice clean way of incorporating those test results into a model via the Bayesian Prior framework. But I'm very curious, Jim, I know that you have some experience with other types of frameworks, how you've done that in the past.

Yeah, no, I like that methodology much better. I'll talk to for example, how it works with Robin, which is probably the most popular open source package out there, that meta put out there for semi autonomous MMM package. And it's great that it actually does allow you to pull in test results to calibrate the model, which not a lot of the open source frameworks do that. And basically what it does is you put in point estimates. So you have a test that ran over, you know, maybe it was a period of one month. You basically say, okay, here's the test that ran. It said that, you know, Google search ROI was 1.63. So that's the point estimate that I'm going to input into the model. And then it basically uses that as a constraint against what the model says. The ROI of search in this example was. So it's not quite as smart is what Michael mentioned with, with what Recast is doing, but it does kind of help constrain what the model is going to say. The, the value of, of a channel is compared to what the the actual test said, you know, how it performed.

Great.

Okay, we've got time for probably one more question, maybe two. So we got a question about holdout forecasting. Is one month a general rule for what to use as the holdout, or should we do something different? This, I don't know, is the answer. I can tell you what we do at Recast. At Recast, we generally look at one month for ongoing monitoring, but also seven days, 60 days and then 90 days and sort of have a rolling check of how well the model is doing over those different time spans. What's like. Right. I think is probably going to vary business to business, depending on, you know, how long are the estimated time shifts for the channels that you're modeling. Right. If you have a much short time shift.