Tom Vladek from Recast explains how upper and lower funnel marketing channels interact in media mix modeling. The video covers causal inference and regression techniques for measuring how awareness campaigns affect conversion channels, and why isolating each channel's true impact is harder than it looks.
Hi, I'm Tom Vladek, co founder and co CEO of Recast, the platform where the most advanced brands measure, forecast and optimize their marketing spend. Today I'm going to talk to you about the interaction between upper funnel and lower funnel channels in the context of a media mix model. Upper funnel channels are things like tv, out of home radio, podcasts. These are the kinds of channels where you are generating awareness, introducing your brand, reactivating customers that may already be aware of you, reminding them that you exist. These channels are typically capped by your budget and your willingness to spend into those channels. Lower funnel channels are things like branded search on Google or in the iOS store. These channels depend on people searching for your branded keywords. They are driven by upper funnel activity. Say you run a TV spot, introduce somebody to your brand, they type your branded name into Google, and they are served a branded search ad. That would be a typical journey that would induce some of this branded search spend. Anytime you have an interaction like this, whether in the context of media mix modeling or elsewhere in statistics, it's important to think through how the system works qualitatively before you start doing any quantitative modeling, as I will show you in the following exercises. If you don't account for this dynamic where your upper funnel spend causes lower funnel spend, your model is almost certainly going to be wrong. The right way to do it is to use a hierarchical approach. For those true Bayesian stats nerds out there, this is going to be a slightly different take on a hierarchical model than you may be thinking, but it's still hierarchical in its approach. I'll walk you through exactly how we arrive at that conclusion and what we do to solve for it. So right now you are staring at a screen of daggity.net Daggity is a play on the acronym directed acyclic graph. One half of the causal inference community loves to use directed acyclic graphs as a tool for causal inference. Because once you set up a dag, you can use some analytics tools to tell you if you've set it up correctly in order to estimate the causal effect that you're interested in. Right now we have a very simple DAG that we'll use today which has three variables. Upper funnel channels, remember, think something like tv. Lower funnel channels think something like branded search and sales. So we're going to start defining these variables. Sales is the outcome that we are interested in. And these all went blue because they are ancestors of the outcome, meaning they are upstream of it. Now, let's define upper funnel channels as My exposure. Great. No open biasing paths for the total effect. Now, if I want the total effect of upper funnel channels and I simply include upper funnel channels in my regression and sales as the dependent variable, I will get the total effect. However, if I adjust for lower funnel channels, meaning if I include them in the regression too, I have a problem. This model will be incorrectly adjusted if I want the total effect of upper funnel channels and I also include lower funnel channels in the model. The problem is, is that some of the effect of upper funnel channels is by causing branded search spend to go up, which then causes some people to purchase the product. So there is some indirect effect here that is no longer going to be correctly assigned to upper funnel channels because they're going to be captured by this lower funnel channels term instead. So that is a problem. Now if I want the direct effect, no problem. I can capture that here. Now what if I don't adjust for lower funnel channels and I leave them unadjusted? Well, if I want the direct effect of upper funnel channels now, I can no longer capture it. Right? If I just include them in the regression and don't adjust for lower funnel channels now, I won't be able to. Right? So the problem is that my regression is only going to give me one term out of the two that I might be interested in, depending on how I adjust for lower funnel channels or not. So this is a problem. How do we get around it? That's what I'll show you next. I have a very long R script here and I am giving away the results. So I'm going to clear that plot. If you saw it and you were able to interpret it in time, kudos to you. Let's set things up here. Okay, so I've loaded some libraries and let's say I want 1,000 time steps. So the first thing I'm going to do is, is I'm going to simulate our predictors which are going to be marketing spend. As in the previous working session, I've done this in a sort of easy way, right? So I've generated a random normal variable. I've taken the cumulative sum. This will be a random walk. Then I've put it through the inverse logit function which will bound it between 0 and 1 and then I've scaled it. When I do that, I get a bounded random walk that will look like this. Right now I'm going to generate some ROIs, direct ROI, capturing the indirect effect, which leads me to capture the total effect, the indirect spend, and then the total spend, which is down here, and then I get a roi. Now, in this particular case, we come off it way wrong. Our naive ROI is 2.52, but the correct ROI is 1.4, much lower. Why is that? The indirect Effect here is 0.86 and the indirect spend is 2.32. Most of the spend here that we have to account for is induced lower funnel spend. This particular channel is really effective on its own, but it's also driving a lot of additional spend into lower funnel channels. That is less efficient on its own. It's very efficient, has a very strong direct effect, but it is driving less efficient spend into lower funnel channels, bringing its overall effect down. If you were to naively estimate or be given the direct effect, you might think, wow, 2.5x, that's great. But the actual total effect is much lower, 1.4x. So let's see what happens when we try to capture these numbers by using a regression. And I'm only going to do ordinary least squares here because we've set up a very simple linear equation here, so it should all work. Right. But let's see what happens when I do that. Okay, so I have this naive regression with all of the channels included, and I'm going to pull out the estimates. You might wonder what this negative one is for. Well, that's the intercept. I don't particularly care about it right now. And so I have the estimated ROIs for x1, for x2 for lower funnel channel 1 and for lower funnel channel 2. And they correspond with the two upper funnel channels that I have in the model and the two lower funnel channels that I have in the model. Well, how do we do? The estimates really line up closely? Right. The estimates and the direct ROIs are very good. We have almost no issue here because we've set up the model, the regression, and it's basically line for line or element for element with the equation that generated the data. But the question is, do we capture the total ROIs? And we don't. This is what we saw earlier. The estimate is 2.5x that total. The actual is 1.43, the region.
