Variables to include in an MMM | Recast Office Hours #1

Foreign.

It is 11:17 my time. So 1:17 for the folks on the east coast and a variety of other times depending on where you all are calling from. But that means that I want to jump in here. So first of all, welcome to the Recast office hours. This is the first one of these that we have ever done. Oh six seventeen in London.

Great.

So we're winding down at the end of the day. There's. This is the first one of these that we've ever done. We were honestly a little bit surprised by the number of people who registered for this. And so we are going to try to do this in a format that we think is going to work as best as we possibly can. But we also plan to do a bunch more of these and we'll iterate a little bit on the format, try to figure out what works and what doesn't as we move on to the next ones. So thank you all for joining. This is going to be a little bit of experiment. Our goal really is to like one have fun. Hopefully we can all like have a good time together and then also learn some really interesting stuff. So that's the plan for today. Quick introductions. Hosts are going to be myself and Tom, we're the co founders of Recast. We both have a lot of passion for causal inference, statistical inference, econometrics and really excited to use this time today to talk about some topics that we just honestly really care a lot about. Ground rules for the session today. So because we have so many people, we want to try to like manage this in a reasonable way. So first things first, please put yourself on mute and stay on mute until slash unless you're called upon. Once we get to like the sort of open discussion Q and A part, we can use the hand raising feature on Zoom, but in general going to try to keep everyone on mute. That way we can make sure that everyone can can hear the content as questions come up. Any questions that you have, please drop them into the Zoom chat. We will be keeping an eye on the Zoom chat for the questions that come up. We will try to feed the most relevant or interesting ones into the conversation. But any questions that we don't cover live, we will try to answer offline and then share out via email. So drop questions that you have into the chat at any point in time so that way we can work off of them and then we'll capture all of them by the end of the call to make sure that questions can get answered or addressed at a later point.

Great.

So the way that this is going to work is we're going to spend about 20 minutes with Tom and I talking through some material that we've prepared on variable selection and using directed acyclic graphs or causal dags for thinking about how to do statistical modeling and getting causal inference out on the other side. At which point we'll transition over to like open Q and A. Hopefully we'll have a bunch of questions in the chat that we'll be able to work off of. But that's the basic run of show. So without further ado, let's jump in. So we were going to talk today about variable selection, particularly in the context of doing some sort of regression analysis. And this could be like an OLS least squares type regression or something more complicated. Almost all of these points are going to generalize across any different type of regression that you might be doing. Key points. So first of all, we're talking about today in a, in a causal inference framework, right? We're working in this world where we have some data and we want to make inferences about some causal relationship that is true out in the real world that we're trying to estimate off of data. This is different than like a pure prediction exercise. Sometimes when people are talking about regression you'll have two people talking like, just like past each other. Well, one person is coming from a machine learning, very prediction oriented background and another person is coming from a causal inference, scientific background. And the goals there are so different that often, you know, different approaches are totally fine. In this case for today, we're really going to be talking about inference coefficients or not coefficients, but just dumping variables into the regression and expecting to get something useful on the other side. So the truth here is that the first regression is actually the most correct, right? Only controlling for the left leg is much better than trying to control for both variables. That's because the true data generating process, which I put the R code into the slide here to show you, is actually just that. Height is a function of the average between the left leg and the right leg. If we sort of know what the true data generating process is, then it's fairly straightforward to build a model. But in general we don't know what the true data generating process is. And we have to make assumptions based on our knowledge, scientific knowledge of the world, and that's where the actual modeling work comes in, is we have to think hard about what is going on in the world, what is the truth, what is the right model of the data that we're looking at and then use an appropriate statistical method to actually estimate that model. So the point that I really want you to take away from this very quick example is that controlling for both leg length here ruins the regression. Terrible idea, bad practice. And in general that should cause you to think about, okay, in general there might be other situations where controlling for more things makes our regression worse. And so what are those situations that might happen? Even in cases where it's not quite as obvious. Right. It's pretty obvious that like the left leg, right leg thing is going to cause us problems if you sit and really think about it for a little while. But there are other situations which aren't quite, quite so obvious but which can crop up when we are trying to do advanced statistical model. Tom, anything to add to any of that before we switch into some of the more marketing related sections of this?

No, I mean I think I can cover some of the like tangents and sequel A to that on the next slide.

Cool. So I'll let you take over from here. Tom.

Sure. Yeah. So I mean, as Michael was saying, you know, regression without thinking about which variables go in. It's just an exercise of if I know this number, can I do a better job of predicting some other number? Right. So if I know affiliate commissions and I, and I don't know sales, will I do a better job of understanding what sales was? It tells you nothing about the causal pathway. Right. And what we're going to go through in the next couple of slides are a couple examples where it might, you might think that you want to control for it might seem very natural to control for a variable like site visits or web traffic when you're trying to predict E commerce revenue. But if you do so, you're going to end up with a very different interpretation of some of the variables you might care about. You might think in a model where I want to predict E commerce revenue, I might use affiliate commissions because affiliate commissions are going to be highly correlated with E commerce revenue. But if you do that, you're going to end up with a very different estimate of your marketing activity than the true one. There's all sorts of examples like this and we'll get into some of them. I hope that you might have some examples in the chat where some of these biasing pathways might come up. But we get questions like this all the time. One that was really common was controlling for Covid. Would it make sense to control for Covid? And I think in some cases it might have been, in some cases it might not have been. But what I want to introduce to you on the next couple of slides are a couple tools for thinking about whether or not it makes sense to introduce a variable into an MMM model. So if you go one more slide, we can start with affiliate commissions. So this is something we get asked all the time. Can we include affiliate commissions or can we include website visits into the mmm? Now what we're showing here is a tool called a directed acyclic graph or a dag. I'm not going to get into all of the math here, but dags allow you to undertake a qualitative exercise of drawing what affects what. Now, without getting into any specific numbers, it allows you to say, okay, we think that marketing activity is going to drive purchases. That's pretty unobjectionable. We also think that organic demand is going to drive purchases. And so in both of those cases we have a node for organic demand and a node for our marketing activity. And then an arrow saying, when I change this thing, it changes that thing. Right? Now we also have this variable called affiliate activity. And affiliate activity is going to drive purchases, right? People have their website or they have their podcast and they're driving people to your site and it's going to drive purchases. But purchases is going to drive commissions because on a commission based affiliate is.

Full on the other side. So the truth here is that the first regression is actually the most correct, right? Only controlling for the left leg is much better than trying to control for both variables. That's because the true data generating process, which I put the R code into the slide here to show you, is actually just that. Height is a function of the average between the left leg and the right leg. If we sort of know what the true data generating process is, then it's fairly straightforward to build a model. But in general we don't know what the true data generating process is. And we have to make assumptions based on our knowledge, scientific knowledge of the world. And that's where the actual modeling work comes in, is we have to think hard about what is going on in the world. What is the truth? What is the right model of the data that we're looking at, and then use an appropriate statistical method to actually estimate that model. So the point that I really want you to take away from this very quick example is that controlling for both leg length here ruins the regression. Terrible idea, bad practice. And in general that should cause you to think about, okay, in general, there might be other situations where controlling for more things makes our regression worse. And so what are those situations that might happen? Even in cases where it's not quite as obvious. Right. It's pretty obvious that like the left leg, right leg thing is going to cause us problems if you sit and really think about it for a little while. But there are other situations which aren't quite so obvious but which can crop up when we are trying to do advanced statistical model. Tom, anything to add to any of that before we switch into some of the more marketing related sections of this?

No, I mean I think I can cover some of the like tangents and sequelae to that on the next slide.

Cool. So I'll let you take over from here, Tom.

Sure. Yeah. So I mean, as Michael was saying, you know, regression without thinking about which variables go in is just an exercise of if I know this number, can I do a better job of predicting some other number? Right. So if I know affiliate commissions and I don't know sales, will I do a better job of understanding what sales was? It tells you nothing about the causal pathway. Right. And what we're going to go through in the next couple of slides are a couple examples where you might think that you want to control for a. It might seem very natural to control for a variable like site visits or web traffic when you're trying to predict E commerce revenue. But if you do so, you're going to end up with a very different interpretation of some of the variables you might care about. You might think in a model where I want to predict E commerce revenue, I might use affiliate commissions because affiliate commissions are going to be highly correlated with E commerce revenue. But if you do that, you're going to end up with a very different estimate of your marketing activity than the true one. There's all sorts of examples like this and we'll get into some of them. I hope that you might have some examples in the chat where some of these biasing pathways might come up. But we get questions like this all the time. One that was really common was controlling for Covid. Would it make sense to control for Covid? And I think in some cases it might have been, in some cases it might not have been. But what I want to introduce to you on the next couple of slides are a couple tools for thinking about whether or not it makes sense to introduce a variable into an MMM model. So if you go one more slide, we can start with affiliate commissions. So this is something we get asked all the time. Can we include affiliate commissions or can we include website visits into the mmm? Now what we're showing here is a tool called a directed acyclic graph or a Dag, I'm not going to get into all of the math here, but dags allow you to undertake a qualitative exercise of drawing what affects what. Now, without getting into any specific numbers, it allows you to say, okay, we think that marketing activity is going to drive purchases. That's pretty unobjectionable. We also think that organic demand is going to drive purchases. And so in both of those cases, we have a node for organic demand and a node for our marketing activity. And then an arrow saying, when I change this thing, it changes that thing, right? Now we also have this variable called affiliate activity. And affiliate activity is going to drive purchases, right? People have their website or they have their podcast and they're driving people to your site and it's going to drive purchases. But purchases is going to drive commissions because in a commission based affiliate program, people, the affiliate only gets paid after a purchase. Now, that's not true for every affiliate program, but in specific ones where you're paying the affiliate based on a purchase they've made, the commission doesn't actually cause the purchase. It's actually the other way around. The purchase causes the commission, right? And so in this particular case, we can see that if you control for commissions, you're actually controlling for a variable that is downstream of the thing that you want to measure, which is its effect on purchases. And if you want to measure the effect of marketing activity, you're going to get a biased effect because the effect of commissions is going to in most cases steal from the effect of marketing activity and you're not going to get a reliable estimate. Now the nice thing about these tools is, is that you don't have to figure it out yourself. You can simply draw the DAG and then say, hey, this is the thing that I care about measuring this is the qualitative set of arrows in terms of this thing affects that thing. And then you can say, are there any biasing pathways? If I control for this, does it cause bias in the thing that I actually care about? Now the tool that we used here is called Daggity. It's online. You can draw these things and you can select things as being adjusted for, meaning it's included in the model, or unadjusted meaning it's omitted. And it'll tell you if it causes a biasing pathway for the thing that you care to measure. In this particular case, it does cause bias because you're adjusting for something after the thing that you care about, which is purchases. If you scroll to the next slide and you do this in Daggity, you get, okay, if I leave affiliate activity unadjusted, it's very happy and says, okay, great. You can get an unbiased estimate of marketing activity on purchases. And what's counterintuitive to this is that you actually have to omit it from the model. You cannot include it in the model. Once you include it in the model, you cause bias and then you get the sort of wrong estimate of marketing activity. And that's what's so pernicious about this case. So I'm going to sort of pause here because we sort of stealthily covered a lot of ground here and I just want to sort of like open up. You can, you know, if you have a question, I guess you could sort of take yourself off mute or quickly type it in the chat. But I just want to make sure that we're not leaving anybody behind and that what we've sort of talked about here is at least making some sense to people. Feel free to just shout. No. If anything is too confusing, get a thumbs up from Mr. Schroeder.

David Schrader gets it. We love that.

Or Schrader. Yeah. Or Schroeder.

Yeah. I don't know. It could be either. He's nodding his head as if. As if we did. As if we had chosen one of the two.

Fellow former Greenpoint resident. Okay, great. And what's. What is so challenging about this, especially as I assume most people on the call are actually involved in building these models, is that stakeholders don't tend to understand why you do this. Right. But can't you get a more predictive model if you include more variables, especially more variables that seem to be so tightly coupled with the outcome? And the problem is, in this particular case, you can't know a priori how many commissions you'll pay. In fact, if you were to include that in a predictive model, you're sort of cheating, right? Because you're including in the predictive model something you would not have known in advance and something you cannot directly control. You can control marketing activity, but you can't control the number of commissions paid, and the commissions paid come after purchases. And so by including it in a model, you're sort of assuming you would know something that you wouldn't have known a priori, and that's looking ahead or allowing future information to leak into your model. And so it's a subtle conversation, but one that you need to have. So we'll plow ahead to the next example, which is website visits. So this is another question we get all the time, which is, hey, do you include site traffic in your model or can you include site traffic in your model? We'd love to, you know, see how that drives purchases. And again, you know, if you put this into Daggity or any tool that can qualitatively analyze DAGs for biasing pathways and you adjust for website visits, meaning you include it in the model, you'll get some alarms going off saying, whoa, whoa. This causes a biasing pathway. And the reason it causes a biasing pathway for the effect of marketing activity is that marketing activity is mediated by website visits. Website visits stand in between marketing activity and purchases, meaning that some of one of the ways that marketing activity causes purchases is by driving people to the website. And so if you control for website visits, what you are essentially doing is you're saying, hey, the entire pathway that marketing activity drives purchases through website visits, we're just subtracting from the marketing activity variable. We're not allocating to it. So, for example, if you had a channel that was super effective at driving people to your site and then super effective at getting the people that went to your site to then purchase, but you were to include website visits in the model, it's very likely that you'd say that channel had no effect. Right? Because what you're saying is all things equal, right? Let's assume what is the effect of this channel if it did not drive website visits up, is what the regression is saying. That's what happens when you control for website visits in this model. And that's why you get a biased estimate of the effect of marketing activity. Now, again, if you sort of go through the exercise, the qualitative exercise of not adjusting for website visits, you get a unbiased pathway. The qualitative tools that evaluate DAG say, this is all really great, you're going to get an unbiased estimate. Now, again, this is a very challenging conversation to have with stakeholders because typically they won't intuitively understand why not including a variable that seems to be so predictive of purchases should be included in the regression. And again, this is one of these things where you can say, well, including website visits in the model is assuming you will have information that you wouldn't have. If website visits are an indicator of marketing activities effectiveness, then we shouldn't include it in the model because we don't know how effective the channel will be at driving website visits. And again, a challenging conversation to have. But, and one of the cases we've been discussing where you actually don't want to include a variable in the model. But the downside is if you do include it, you're going to get very biased estimates of the effect of marketing.

We've got some questions coming in the chat, but I want to just wrap up this section really quickly. So while I'm talking a little bit about some of these other resources, Tom, maybe you can read some of these questions and start thinking about your answer. So we're going to put some links to these additional resources in the chat here for you all to check out. These are four of our favorites internally here at Recast for thinking about. First of all, doing causal inference using DAGs and then using statistical models to try to learn things about the world. So the first is Statistical Rethinking by Richard McElrath. He has an amazing book, it's on the shelf behind me here. And he also has an amazing free YouTube course which is just really good. He's really funny, really engaging, amazing lectures. That it's all available for free is just an incredible resource. So highly recommend checking that out. A crash course in good and bad controls. This one has Judea Pearl as the third author. He's like a big deal in causal inference. It talks through a bunch of these examples of places where people go wrong doing something that seems intuitive, like, oh yeah, let's control for this thing. But it actually causes problems. And causal inference. Really, really good resource. Opened my eyes personally to like see this and see all of these examples. And the last two are just other really good resources on causal inference. They both deal with dags. They both deal with different statistical estimation proced. The first one is called Causal Inference the Mixtape, and the second one is another free book called the Effect. Highly recommend checking all of these out. All right, I have stalled for time while Tom has read some chats here, so I'm gonna stop sharing my screen and we'll let Tom maybe take a stab at one or two of these questions.

Yeah, sure. And I'm just gonna say there's been some questions that we got prior to this that I'll probably come back to, but I'm going to start with Sebastian's question, which is would we replace website visits by assumptions for each channel, such as the conversion rate per channel of the funnel based on website visits, retroactively speaking? So, Sebastian, I'm sort of assuming that you have metadata on the channel, right? So you're saying, hey, we can see that people from this channel tend to convert at 5% and people from this other channel tend to convert at 4%. So it is in principle possible, in my opinion, to. Well, first of all, Sebastian, I saw you came off, you know, no video. So maybe if you want to clarify that question or add a little detail, now would be a good time.

Yeah, great.

So basically, we have assumptions. We know how each one of the channels converts. So based on the first things you said when we started this meeting, do we want to add assumptions on each.

One of the channels?

Maybe not on a user level or user interaction, or trying them all to understand based on more features how or predict how the interaction will flow, but more on how those channels on aggregated level work?

Yeah, I think that. So just, you know, we don't do that, Recast doesn't do that. And one of the reasons we've done it, we've not done it, is that we found the number of assumptions that you have to make to include those sorts of variables is very high. So for one, you have to assume that whatever system that you're using to tag people as coming from a channel is accurate. You know, they may have come from a last click, right? But they might also have seen other things. And so they may have seen a billboard, they may have seen another ad on another channel before coming in. And so that sort of data can be a little bit challenging to work with. The other is that you're sort of working with a qualitative setup where you have, let me sort of draw some. A quick DAG here where you have channel A spend and channel A effect as two nodes in the variable as two nodes in your dag. And the right way to include those sorts of variables as they would be upstream of this effect node. So you'd say struggling a little bit to move this around. So you may have like conversion rate as pointing to the channel effect. You might have like, you know, value per conversion. So let's say in a world, there might be a world where you had conversion rate and value per conversion as on a channel level basis. And you know, there may be like a CPM.