Wharton School's Ryan Dew: Why your marketing mix model (MMM) is broken | Coffee Breaks

Ryan Dew is an assistant professor of Marketing at the Wharton School at the University of Pennsylvania. Ryan's research focuses on how machine learning and Bayesian statistical methodologies can solve real world marketing problems, helping marketers make smarter data driven decisions. In today's episode, we had a really wide ranging, fascinating conversation touching on Bayesian versus frequentist statistical philosophies, how marketers should really think about and appreciate uncertainty, the role of assumptions, identifiability and correlation versus causation in media mix modeling, and how different types of models might be conflated in certain situations. There's a whole lot more in the conversation and Ryan was just a really, really wonderful guest, getting both into the deep technical weeds, but also keeping it very accessible. I hope you really enjoyed the conversation. I'm excited for, really excited for this conversation. I've been looking forward to this one for a very long time. The first question that I wanted you to sort of comment on is that, or maybe the first topic. I see a lot of debate, some of it is frustrating online about Bayesian versus frequent versus frequentist approaches to lots of different things, but particularly in the field of media mix modeling, people seem to get very worked up over this. Given your background, your expertise in statistics and marketing science, I'd love for you to talk a little bit about, like what do people mean when, when they talk about Bayesian versus Frequentist approaches? How do you think about where that matters? Where does it not matter? Like what should marketers be thinking when they're like hearing different vendors or zealots talking about Bayesian versus Frequentist approaches in this domain?

Sure. Yeah. Well, by sort of background, I come from this sort of Bayesian statistical perspective. And so I am a non neutral participant in this, in this conversation. But I'll try to give the, I'll try to give the sort of the sort of lowdown with minimal philosophical bias. So kind of just like from a fundamental perspective, what people are really talking about is they're talking about different ways of thinking about uncertainty in the models that we're building. And so maybe it's even, maybe it's even sort of useful to start by thinking about what does uncertainty even mean and why would somebody in marketing care about uncertainty? And so I think sort of fundamentally when I say uncertainty, I do mean kind of intuitively what you think of when I say uncertain. Like the future is uncertain or the effectiveness of what we're doing, what we're putting our dollars into is uncertain. And so the question is, how do we quantify, how do we think about that uncertainty and what it means and how do we make decisions based on that uncertainty. And I think this is sort of the philosophical difference between sort of the Bayesian worldview and the frequentist worldview is sort of thinking about what does uncertainty mean and how do we quantify it in a, in a statistical model, like, like a media mixed model or something like that. And so just like sort of right off the bat, the, this sort of being sort of dogmatically Bayesian versus dogmatically frequentist, I don't necessarily think this is the way to be thinking per se, although I again, I tend to fall on one side of the fence more than the other. I think instead you can kind of ask yourself what view on uncertainty and what kinds of models make sense in the context that I care about. And I think from that perspective, the Bayesian perspective, the Bayesian worldview makes a lot of sense in the context of, of, of mmms. And I'll, I'll explain sort of why that, that why I think that way. So the, the, the gist of it is that when we think about uncertainty from a Bayesian perspective, it is something that really, it, it sort of, once you have the model in hand, the uncertainty kind of just falls out from that model. So you've got your set of sort of assumptions there. You take those assumptions, you combine them with data and you get some estimates of how effective is this marketing channel or what does the future look like if I spend X? And all of that kind of falls out from, from the model and the assumptions that you put down on paper in, or probably not on paper actually, but you know, in whatever, you know, code that you're, you're using. From the frequentist perspective, the idea of uncertainty is a little different. It is often thinking about, say, how would the, if I could hypothetically go back and get a new sample of data and refit this model again and again and again, how would the parameters of the model change given these sort of independent draws of these sort of hypothetical random data sets? And very often this notion of uncertainty is also sort of thinking about kind of limiting cases. Like in this world where I had tons and tons and tons of data, sort of my uncertainty would look like this. And I think from a marketing mixed models perspective or media mixed models perspective, or maybe I should just say mmm from from now on to avoid needing to say so many syllables. Very often we're kind of in this world where we might not really have that much data and we're in a world where sort of we want to be able to think about assumptions, inspect those assumptions, and maybe even use those assumptions sort of strategically to improve kind of the estimates that we're getting. And so I think from that perspective, the Bayesian approach here to thinking about uncertainty, where we're not relying on this idea that we're going to need to sort of repeatedly sample this world and have tons and tons of data, it just makes a lot of sense. I think the cost, though, if you're thinking about, okay, sort of like, you know, you know, we started out by saying there's this debate, surely there's another side to this debate. There is a cost. The cost is in the context of, of what are called, called priors. And so I think priors maybe historically is sort of like this dirty word almost, but I don't think of priors as being inherently a problem or a dirty thing. It is a part of building a model. And so what a prior is, is it's basically a way of thinking about sort of. I come into my model and there's a parameter in the model. Again, maybe think about it like, how effective is every dollar that I spend in this particular channel? Or an elasticity even, even better. So that's a kind of like a part, a variable, a parameter in my, in my model that I'm trying to figure out from the data. In a Bayesian model, you're going to have to specify a prior for that. And the prior is just a way of saying, okay, I come to this model with some sort of knowledge about the value of this parameter. Now, when I say some amount of knowledge that can be, literally, I have no knowledge, like, the prior is wide open, I don't know anything. Or it can be something that I've done in the past, say an experiment that I ran, or just sort of some common sense suggests, yeah, you know, the value of this parameter is likely in this particular range. And so in some sense this can be a good thing. We can use this knowledge, that we have this prior knowledge to kind of build a model that's sort of already pointed in the right direction, that is maybe also being calibrated by expertise and outside sources of information, like experimentation. We can also use this structure of priors to think about sharing information across marketing channels. So it's like, you know, we, we know we don't know much about this channel, but we know something about this other channel and kind of incorporate that information in that way. But with this sort of, with this sort of, with these sort of nice properties, there also comes a risk of Abuse. And I think this is where maybe debate comes in, where there's sort of where people get skeptical. So just like I said a second ago, where we can come into the model and we can say, okay, you know, the effectiveness of this channel based on all of the data, you know, based on the experiments we ran before, the effectiveness is likely to be in this range. So too, can you go to an MMM vendor and be like, yeah, you know, our team all really thinks our social media strategy is really good. And then MMM vendor can say, okay, sure, let me set up the priors in my model such that my results are going to say, your social channel's really good, because clearly that's what you want to hear. That can be completely independent of what the data are actually suggesting. When you actually go and you get that effectiveness estimate or you get that forecast or whatever, it's going to be driven by the data and the prior. And so if that prior is really strong, it's going to influence those things. It's going to say, hey, yeah, your social media channel is really strong because the prior you set says social media is an amazing channel. Without the data could have been anything. The data could have been saying that you're getting zero from it, and the model will still tell you that. And so I think this is where there's a risk, so to speak. And I think what people need to be aware of is that you need to inspect these choices. You need to think about these assumptions rigorously. You need to be working with a, with a vendor or building your own model where you're able to inspect these choices, where you're able to kind of sort of think about the robustness of the model, of these choices, and make sure that these, these sort of choices are being made. Well, when you do that, I think the upsides of the Bayesian perspective really outweigh any of the, any of the potential costs. I mean, other costs might be, it's a little harder to do, you know, a little more computationally complex. But I think ultimately when, when push comes to shove that that complexity is clearly worth it for the benefits that you get from being able to specify these models and get these reliable uncertainty estimates. As long as you're kind of aware of that, of that, of that. Gotcha.

So this was an incredible answer, incredibly thorough from top to bottom. For the sake of completeness, you talked about using priors to bias the model results in some direction or other that isn't really consistent with what the data say necessarily. Again, just for the sake of completeness are the frequentist approaches immune to that in some way? Or like, why would that be different between a frequent frequentist and a Bayesian approach?

Yeah, so, so we're using sort of the word frequentist here to kind of summarize a lot of other things that one could do. So if we're thinking about like what would an. Part of the reason why I like the Bayesian approach too is that is again, I feel it's very transparent, right. Like you, when you write down the model, you have to say, okay, this is the model likelihood, meaning the model itself, kind of like how do the parameters relate to the data that I'm seeing? And then there's these priorities. When you're thinking about alternatives to it, there's usually just the first part. So if I'm thinking about sort of what's the most standard model would be sort of like I have this sort of set of equations, if you will, that sort of relate, okay, the X variables that I've got like my spending to how I think why is going to look and there's going to be parameters like that. That part is kind of going to be going to be the same. This would be, I think it's kind of the more standard way of, of would be the non Bayesian sort of standard way of setting up a model like this. And then the question is sort of, okay, how do you figure out what the best values of those parameters are and how do you think about how uncertain those parameters are? And so one thing that you can do is you can just say I'm going to let the data completely speak and sort of that would be kind of the standard, what you might think of as sort of like the maximum likelihood estimate. Or if you come from a machine learning perspective, you know, you define these loss functions and you minimize these loss functions to get kind of the best predictions of the future that you can get. But they're all going to kind of arrive at some sort of point estimate. And that point estimate in the sort of most basic case is going to be purely data driven. And so in that sense it might avoid this uncertainty about what's going on in the prior. But in many cases that's not necessarily what people will always do. So for instance, if you come in and you say, okay, you know, we are going, we really think that this is the effectiveness of the social channel and I'm approaching this from a non Bayesian perspective, I might constrain that optimization in some way to say like, okay, we're going to find the Best parameter estimates such that this value lives in this particular range or something like that. I might also apply some form of what is called regularization to the model. So basically these kinds of, not necessarily constraints, but these, these assumptions that sort of encourage the model to find values of parameters that live in a certain, in a certain certain range again. And what's interesting about that is that often actually those types of regularization that you're assuming can be mapped clearly onto a sort of assumption evasion model. And what's nice, I think is that actually it's clearer, at least in my view to think about that in the Bayesian perspective from this idea of like this is kind of the prior information we're bringing, whereas the sort of regularization ideas can sometimes feel a bit arbitrary. Like why do we assume this particular sort of thing in our, in, in, in our optimization procedure? I think these would all be sort of very data driven ways of, of, of specifying a non Bayesian model. You can also specify non Bayesian. I mean, you know, like, you know, non Bayesian models encompass a very wide range of things. And so even just sort of an idea where we would just sort of kind of pick parameters, right? Just like sort of, that would be the, almost the most extreme kind of case of saying, you know, we have a prior that's really strong. In a frequentist perspective or just a non Bayesian perspective, we might literally just say, I'm going to assume that this parameter is to. For, you know, maybe you have a reason, maybe you don't. The beauty of the Bayesian approach is that all of these things, this kind of assumptions, the strength of these assumptions is really nicely encapsulated through this mechanism of the prior, which is why I find it much more sort of transparent than the myriad other ways that we could make assumptions about the parameters of our models.

Again, just incredibly clear. I think the way that I sum up this point of view is that with these, especially in the world of more complex models, there are less complex models where I think the difference, there's fewer differences and they matter a lot less. In the world of these more complex models of media mixed models, you have to make assumptions, right? In order to get identifiability, in order for the models to work, there have to be assumptions. And so often the difference in these different approaches is where are the assumptions made and how transparent they are. And so that I think is a little bit of a better and more helpful framing to think about. The potential trade offs of these different approaches is the assumptions have to Go somewhere, there's no getting away from it. It's where are they being made, how are they being applied, and how transparently can we introspect them and try to understand how they might be impacting the outside results? And I think that's where the real trade offs start to come in.

And I think also about how rigid the assumptions are. I think the beauty of the Bayesian perspective and this idea of priors is that we can have this idea that sort of like we expect this parameter to look like this. We expect this, you know, this is the, this is how much this sort of channel is going to exhibit diminishing returns. You know, how much less and less am I getting from every dollar spent? And I have some expectation of that, but I'm willing to sort of admit that that expectation might be a little bit wrong and the model can kind of deviate away from that without imposing sort of like, no, has to be the number two, you know, that's it, you know, And I think that's the sort of fourth point I would add to the list.

You recently published a paper with Nicolas Padilla, very provocatively titled, you, mmm, is broken. I have a number of questions that I want to sort of dig into on this paper that a lot of people online were talking about. I think it was a really, really amazing paper. But maybe to start, can you just sort of summarize high level, what was that piece of research about? What did you find?

Sure. Well, let me, let me, let me put two, two disclaimers right at the top. So first of all, first one's not really a disclaimer. We have a third co author, Anya Shakina, who's an amazing PhD student here at, here at the Wharton School. So, you know, she deserves just as much credit as me and Nicholas. The other disclaimer I'll say is that this is more of a preprint than published research. So, you know, you can still find it on the Internet, but it's sort of in the peer review process as we, as we speak. The summary, though, is so what we were. So it's kind of interesting. I mean, actually maybe an interesting starting point would be thinking about how the paper came to be because it kind of also encapsulates sort of what the paper is about. So I was working, I've been trying anyway to work with a number of different companies on sort of thinking about the science of building marketing, mix models and the assumptions that go into these models. And I was sort of thinking about ways that we can make These models sort of more dynamic. And when I say dynamic, I mean basically this idea that like, if we're going to observe a market for any period of time, things are going to change in that market, right? So we're selling shoes, right? If, you know, we're Nike and Adidas is advertising a ton and then they stop advertising, the value of our dollars might go further or less, you know, so the market will evolve. And so we wanted to think about how we could kind of take a model and sort of incorporate that sort of evolution that happens in a, in a principled, in a principled way. So obviously one way you could do it is just like first estimate the model in this year, then shift the year over a little bit, and then shift the year over a little bit and then naturally your parameters will evolve. But we were trying to think about ways that we could, we could do better and also try to incorporate some experimental data and what have you. As I was sort of fitting these models though, we started to think more critically about the assumptions that we were making. So what we were typically doing is we were kind of assuming very simple relationships between marketing spend and how much sort of revenue we were going to get. And so, you know, these are sort of standard assumptions historically and even in sort of modern things. We were just kind of saying, yeah, let's kind of assume that sort of, there's just this real simple, you know, log law relationship or linear relationship between these, these two things. And as I started pushing on sort of that, that assumption a little bit, I realized that as I sort of changed that assumption that I was making there, the way that sort of the time varying inferences I was making, like as to how this market is changing over time would also change. And I started thinking about, okay, like what's going on here? Like why, why when I made different assumptions about this, this kind of seemingly two independent things, is, is the other one changing too? And so the, the heart of the paper then is sort of our attempt to examine this issue, which is basically to what extent, if we're just in a kind of your standard marketing mix model setting or media mix model setting, to what extent can we really identify, I guess, to use that, that word, we can talk about what that means in a second. But to, to what extent can we really be sure that the story that we are telling about either sort of the shape of our model, if you will, like how does X relate to Y? How does spend relate to revenue? Is there a sort of saturation sort of thing going on where we're getting less and less. Is there a point where we need to spend above this threshold to get. To get more and more? How exactly like does. Do. Does do returns accelerate once we're past that point? All of these sorts of things. So I mean, by the shape, to what extent can we actually figure out the right shape of our model? And to what extent can we figure out how much is sort of are the parameters that we care about, to what extent can we figure out how those things might be changing over time? And the sort of end result, if you will, the sort of thesis of the paper is in many times we can't. In many times you can have two sets of assumptions. One is that I have a sort of simple model and it's changing over time. And then another is I have a kind of complex model and it's not changing over time. And they can both fit the data equally well. They can both, assuming you don't change how you're spending, they can both predict equally well. So I kind of hold out some to do the usual sort of holdout forecast task that a lot of data analysts might do. And I sort of use that as the basis of fully good. In fact, one of the kind of more interesting parts of the paper is we will generate some data with one set of assumptions, meaning. So we know in this data set that this is the set of assumptions that is the truth that generated this data, then we can fit a totally wrong model and we can still get even, sometimes even better fit than the true model itself. And this is. This is sort of a problem. It's a problem in the sense that we have two sets of assumptions. They both, by its sort of standard metrics, seem like they explain the data that we have, and yet they're making different assumptions. Which means that if we then go and use these models as the basis for optimizing a media mix, we might end up with different results. And these results can actually be substantially different from one another because again, they're making different. They're making different assumptions. And so that is, in an essence, the paper. And then we try to think about, okay, is there anything we can do to help this? And we propose some solutions that people might do to kind of say, okay, if I think this is going on, here's how we might run some experiments, some incrementality tests to kind of try to pin down some of these parameters or these assumptions more accurately.

A really amazing explanation. I love the paper. I love the idea of trying to think really hard about what assumptions are we. Making. And what does that imply about how we can use these models? You talk a little bit about identifiability, and so maybe just to step back, can you give a. And I have always struggled with trying to give a good explanation of this topic, so I'm hoping you can do better. What does identifiability mean in the statistical context? What are we trying to do when we're talking about identifiability of different parameters or of different modeling approaches?

Yeah. So I mean, part of the confusion is that people will use this word to mean different things. Right. So I think fundamentally, like the core of the word identifiability is talking about sort of, given this data, can I get sort of the correct answer? Okay, like from. From. From this. From this data? So in our context, what that literally means is sort of given the marks data that we have, can we figure out which set of assumptions is the right assumption without just assuming? Right. So can the data tell us something? Can the data say, yes, this is the right set of assumptions? In other contexts, people. So that's sort of what we mean here. And we're talking about sort of model form identification or identifiability. In other contexts, people might use this word to mean. For instance, you know, I have one class of model I'm thinking of. This is the models that I care about. These sort of. Say they're these nonlinear models where we're modeling, you know, X and Y with some kind of hill function, which is kind of a standard assumption that's used in MMM literature all the time, where there would be a few parameters that kind of determine how does spending relate to revenue? In terms of those same ideas that I was just talking about a second ago, like, how much money do I have to spend before I see some returns? How quickly do those returns level off in many cases, those kinds of parameters. So here we're just fixing the model to be the same. Those kinds of parameters might also be what we might call weakly or not identified. And what that means is it's similar to the idea. I was just saying. I mean, the spirit of the idea is the same, hence us using the same word. But the idea is sort of like there are many different values of those parameters that would give kind of the same relationship between how spending and revenue work. And that's just because we've assumed this really complex sort of function. I mean, it's not really complex, but we've assumed this moderately complex function. And different values of the parameters can lead to more or less similar shapes of that function. When you combine this with the idea that often marketing data, and especially I think MMM data is often pretty noisy, we're then going to be kind of saying, okay, figure out this really nuanced differences in sort of the response given really noisy data. And you're just not going to be able to do it. You're going to find that many different values will give you roughly the same fit. And this is sort of similar to what I was talking about before in the sense that we're saying different models will give you sort of the same fit, but it's in a different context here. We're even talking just within the same model. You have a set of parameters. They may be different parameters that give the same fit. And then there's a, there's a third type which is not necessarily, it's also again, very much related to what we're talking about, but it's sort of a third way people use these words which is sort of about causality and causation. Right. So often when we're talking about marketing mixed models, media mixed models, what we really care about is basically if I'm going to increase spend on this channel by $1, probably you're not going to spend $1, but say, you know, $100,000, then you know, how much is revenue going to change? And that is fundamentally a causal question, right? If I, I'm saying I'm going to do something and intervene and spend more money and then I want to know how is, how is revenue going to respond to that? Is my increase of $100,000 going to cause revenue to change? If it's not causing, if it's just, you know, then we don't care, right? We only care about what our money's causing to happen in the world. And so in many cases people use this word identification also mean can we figure out a causal story from the data that we have? And this is again a tricky issue because as maybe I will assume people know, sort of causation and correlation are not the same thing. Oftentimes there can be an association between two variables without seeing that one causes the other without knowing.

Amazing. So we've got sort of three related but overlapping definitions, all I think really highly relevant when we're talking about doing media mixed modeling and practice. I'd love to dig in because I found that in industry, this sort of technique of using simulation to think about identifiability isn't actually that well known or as used as much as I think it could Be could you sort of talk about the process of simulating data, recovering parameters and then using that to think about identifiability, like what does that look like in practice? How does that work so that our listeners can, can understand a little bit more about how you think about doing that in your work.

So the idea kind of behind simulation methods for this would be sort of in reality we never know what's the true model, right? You know, there's this famous box quote, you know, all models are wrong, some are useful, right? We never know which. What's the, what's the, what's the true, the true model. And so that's not though going to be the case if we are simulating data. So you can imagine a sort of simulation as kind of like this best case scenario that's just never going to happen where we know this is the true model, right? So this is sort of the, you know, if we're the marketing mix gods, we know that this is the correct response function and this is the correct data generating process. And so what we can do then in this sort of, under this sort of set of this best case world, we can generate data, a data set or data sets, changing sort of the parameters. And again, the benefit here is going to be that we know exactly the process that generated that data. Then we can kind of pretend like we didn't, right? We can pretend that we didn't know what was the truth, which is going to be reality, right? When we, when we go to, like I said, just like I go to, you know, fit some data, some a model to some data, we don't know what's the truth. We can pretend we don't know what's the truth. And then we can fit the models that we're thinking about, whether that be the correct model or another model that we're thinking about. And we can then look and see what comes out, right? So what we would hope comes out is, we would hope what comes out thinking about the first two types of identification now, the sort of model form identification and the sort of parameter identification stories. We would hope what comes out is that if we test a bunch of other models, they don't fit as well as the true model. And we would hope that when we look at the parameters that we sort of recovered that those parameters are similar to the parameters that generated the data. Because again, in this case we do know what generated data, we're just pretending we don't. In that case we have a, well, at least statistically, a well identified model, right? When we take that data, we are able to sort of recover a unique model from that, from that kind of data. Issues come up when that doesn't happen. Where for instance in our paper, what we do is we change to a completely different type of model, we fit it, and then we say, wow, actually by all the standard metrics you could never tell these two models apart. Or in the sort of parametric identification issue you might say, oh, the data was generated by a hill function that has parameters 1, 2 and 3. And now we see that this says the parameters are, you know, 125 and 6. And then you're like, okay, well so clearly something's up here that will often happen. For instance, the model is sort of too complex to support the types of data that you have. So you're generating sort of simple data, which by the way, I should stress marketing models are often built on simple data and then you're fitting something fairly complex to it. Complex models can often sort of, you know, different parameters can often generate very similar looking data sets. When the data is really, really simple. In some cases you don't care. But when you're thinking about doing optimization or intervention, often actually kind of figuring out what's the, what's the right model does, does matter.

I love it. I think this is an amazing technique and I think it is underutilized in industry across lots of different applications. Like not just in. Mmm. I think anytime you're working with any sort of algorithm that's even just new to you, like going through and doing simulation and parameter recovery exercises are a really powerful way to start understanding that model and understanding its limitations before you actually put it into practice. I think like if just taking data and running it through a model and being like, yeah, the results seem reasonable is very fool prone, we will say. And so being a little bit more rigorous about thinking about how to introspect these models I think is really good and this is a really amazing technique to do it.

And I would also say like on that point, that even the very act of generating data from an assumed model and just seeing what it looks like can often be an insightful exercise even if you don't go through the second step. So the process that we were talking about before has those two steps. You generate the data and then you try to fit the model that you're thinking about to it. Even just that first step of generating data using different parameter combinations and seeing like what does that data set look like, that's a good way of making sure that the assumptions that you're making about your model actually make sense, right? Very often you might have missed something, you might be making some assumption and then realize all the simulated data sets that I have are assuming that I don't know that revenue is going to be 10 times more than it's actually going to be in any of the data sets that we work with or something like that, or 10 times less or you know, they're all assuming that, you know, the, each channel has a positive return where they might not. These, these sorts of things can, can show up just by generating data from your, your assumed model class and just like thinking about it to make some summary statistics of it. Do some, do some, make some plots. See see what like common sense things is this model or what, what kinds of things are baked into this, into this model? In the insert of the technical sort of jargon, this might be called what's a prior predictive check. If you're, you have a Bayesian model and I think it's a useful way to start with anything and then go to the, the actual, fit the model and see, you know, can you recover the things, the sort of identification checks. I do these things for every model that I, that I, that I ever built, right. This is always the starting point.

I'll share a little story. I came to Bayesian statistics through like a weird circuitous route. Like I started out doing frequentist regressions in Stata and read the Stata and the SAS manual really deeply to try to figure out what magical incantations to apply to some data set to get something out on the other side. And at some point I got like really worried about my lack of understanding of what was happening in the black box. And my way to deal with that was to start simulating data and then applying the methods and then seeing what came out on the other side in order to really try to understand what was happening. Not via the math and the symbol manipulation, which I'm not particularly good at, but I am quite good at writing code and generating data. And once I started doing that more and more and using the simulation to understand the statistics, the transition to using Bayesian methods, which often rely on first specifying a data generating process and then analyzing that statistically, felt very, very natural. And so again, any sort of statistics that you're doing, this simulation method works really, really well and especially well actually in more black box methods that you might be applying. Simulating the data, see what happens, do you get the results you expect on the other side is just a really, really powerful way to, to learn more about the statistics that you're doing and really test the methods and say like, okay, in what situations does it work versus where doesn't it work? And that's a really like, I think, important thing to know when you're trying to use these methods in practice.

Absolutely. I mean, in some sense with computers, a lot of, a lot of math is actually not that important. Of course, like, you know, you want to write an academic paper about how a model works, I guess you better understand the math. But at the end of the day, these sort of simulation based techniques can really just sort of substitute for, for that entirely. Right. This is, I mean this, this has sort of been the backbone of a lot of what people we call like computer age statistics, which sounds very old school, but there's, I think there's a book like, with that, with that title, that's a pretty interesting, it's a pretty interesting read. But just in general, like the, the whole idea that sort of, even if we can't do the. Almost all Bayesian statistics is based on the idea that like there's a lot of math that we can't do. Right. Like when we want to compute these uncertainty estimates, a lot of times we couldn't do it. Like if you sat down with the math, you couldn't do it. But a computer can simulate it and you can figure out things based on those simulations that, that, that will give you sort of the answer that you need. And so absolutely, that's a, that's a great backstory too.

I want to sort of wrap up the discussion on this particular paper. You know, you had mentioned, look, you know, we simulated data, we tested it in a bunch of different ways. I would love to hear a little bit more about like again, in a little bit more detail what the implications of that were of some of what you found. And then you also mentioned pointing a way forward to how might we either run experiments to pin down these parameters or other checks that we might run. I would love to hear you voiceover what those recommendations are.

Yeah, sure. So at the end of the day, so the paper is kind of divided into chunks. Right. So the first chunk is more or less what I already already discussed, which is that kind of in the modern and I mean the sort of typical mmm data and the typical models we're fitting, you have this fundamental identification problem. The issue is that you want to know which one's right or maybe even which one is sort of more right. I mean, in some sense you might expect both frankly to be operating I think it makes sense that marketing exhibits a nonlinear relationship with revenue. Right. Surely you can't just continue to pump dollars into sponsored search ads and continue to get sort of some kind of value out of that, you know, at the, at the limit of, you know, spending, you know, billions of dollars on Google. Google will be very happy, but you're probably not going to get much of a return out of that. Simultaneously, I think intuitively, you know, in many markets, there are probably changes that happen in that market, especially if you're looking at this over time. And so I think that the question is sort of first, the first order question is just, okay, in some context, maybe we care about one more than the other, or maybe we think one of these forces is more relevant than the other. And so we want to figure out kind of what's the first order concern. Should we be thinking about making some kind of simple assumption about the nonlinearities, the sort of the shape of our model, and then being very flexible when it comes to estimating how those parameters change, how that relationship has changed over time, or should we do the opposite? Should we be very flexible about kind of the relationship between X and Y, but maybe assume away those dynamics, maybe they're not super, super important in this. In this case, you could also imagine a world where you're kind of somewhere in between, but that's a very complex world to be in that we sort of skirt around in the paper. And so the question is, if you wanted to figure out which of the two is the most relevant force, is there anything that you can, is there anything that you can do? So one thing that you can do is you can kind of think about is my data, the type of data that is making this more likely. This what we call conflation, meaning the lack of ability to tell which one's the right model, is my data putting me at a higher risk of this conflation happening. So again, the way that we sort of define conflation is doing what we think anyway is sort of the most basic way that, say, you're kind of running the mill data scientists would approach this topic, which is, I'm going to see if the model predicts well on data that it hasn't seen before. So this sort of holdout validation, and we say the models are conflated if they give essentially equal performance in that holdout, holdout validation, when we know what's the true model, we define it as the wrong model does, as well as the true model. When we don't know, we just say that they do the same. And so of course, one way you could check if the models are conflated is just to do that test. I mean, hold out some data, fit the model, see how the predictions look. If you make these two kind of fundamentally different assumptions and the models give the same fit in that holdout period, then you're clearly in this conflation situation. But what we also discussed in the paper, that there are some kind of properties of the data generating process that make this more likely to happen. One of these properties, and this is sort of where math comes in and I don't, I don't want to go down the rabbit hole too much. The paper is out there if you're curious about some of the details. But some of the things in the data that can make it more likely to happen are things that you don't really have much control over. So for instance, the noisier the data is, the harder it is to pin down any model. Doesn't matter if it's time varying, nonlinear or whatnot. So the noisier your data is, the hard, the more likely conflation is to, to happen. Another thing that's a little bit more interesting though is that conflation becomes more likely when the sort of spend variables, the X variables in your model are related to each other. Meaning, for instance, suppose that I am always setting my ad spend as some function of the previous month, quarter, week, whatever. So I say I'm going to set it to be equal or I'm going to set it to be 1% more. Something like this. We show that in these cases where there's this sort of very highly, what we would call more technically kind of autoregressive component to spending, that this conflation is more likely to happen. And that's something that you can kind of check. I mean, you can first ask yourself, is that how we set our ad spending? But in other, you could actually just go to your data and think about sort of, okay, how sort of related are my X's over time? When they're more related over time, the more likely this conflation issue is to appear. That, by the way, I'm not saying is a bad thing or a good thing. I'm not making a value judgment about sort of should X's be related over time. It's more of a kind of mathematical, mechanical thing that when they are more related over time, this conflation issue is more likely to appear. Final thing that comes up very much related to that is that very often people will induce this kind of smoothness in the X's by applying transformations like ad stock. So Adstock, I would imagine many people who would be watching a video like this would already know what it is. But ad stock is a way of thinking about kind of the carryover effects of spending. So if I spend money today on some channel, maybe it has an effect today, but more likely it's going to have an effect sort of next month and the month after, and the month after too. It's not necessarily going to be an instantaneous effect because if you're in an industry or you're advertising in channels where you expect there to be some kind of carryover effect, and you're applying these ad stock transformations to the X's in your model, then this conflation issue is also more likely to crop up because you induce smoothing in your excess. And so this comes then to kind of the sort of thing. So we point out this problem potentially exists. The question then is like, can we do anything about it? And the answer is yes. The answer kind of comes back to something that people I think should be doing already, which is you can run experiments, certain types of experiments that can help you not just figure out sort of what's the value of a channel, but certain types of experiments that also kind of help you think about which model makes more sense for your data. And I'll give kind of a couple intuitions for sort of how this might work. So one way that this experimentation might work, which is not really the focus of the paper, but it's sort of just a sort of side note, is that the, the, as I said before, this conflation issue is a problem under intervention. What I mean is that if you're going to use these models to optimize your spending, that means you're deviating from kind of current spending patterns. That's when these models will give different predictions. And so if you are kind of constantly running sort of sort of spend experiments, incrementality tests, these kinds of lift test, whatever, whatever of the many words that refer to kind of the same thing, whichever one of those you're, you're doing, and you're doing that in kind of say the future, and say maybe this holdout period, then having that kind of intervention in the future, you can kind of see which model does a better job of predicting under that new data regime. And that I think is a, is an important thing because these models really give kind of equal predictions under the status quo, not under intervention. That's kind of the crux of the whole thing. But more than that, what we do in the paper is we actually think about sort of, okay, if really your goal was just to figure out sort of which effect is most relevant, is it that sort of effects are changing over time or is it that there's this nonlinear relationship between X and Y? You can actually design tests that do a good job of doing, getting the answer to that question. And the way that you do that. We propose kind of two ways to do that. One a little bit more sophisticated, one a little bit easier to implement. Let's start with the easier to implement one. So the easier to implement. One kind of makes this observation that kind of. If you have sort of time varying effects, what that sort of means is that if you say you're sitting here today and your spending is X on, on some channel and then you fast forward a year, two years or six months and you again spend X, those sort of marginal return at X should be kind of the same at those two points in time. Sorry, sorry. It should be different for the. Sorry, for the time variant. They should be different at those two points in time. Right. That's kind of the definition. On the other hand, if there's not really. So if sort of non linearities are sort of the thing that's of interest, then what we're really talking about is regardless of time, if you go from X1 to X2, you should see sort of the same in. You should see this increase. And if you do that today versus you do that six, six months from now or a year from now, that sort of marginal increase should be kind of the same. And so what we propose is, we propose this sort of simple test which we call the seesaw test. And the seesaw test is basically saying you can fix some period of time, a very short period of time, say one month, one week, two weeks, whatever. And you can vary spending from some level to another level and do this kind of over and over and over. Sort of like these bump up incrementality tests that a lot of companies do. And you can observe kind of what happens in that period of time. Depending on sort of the sort of. The time varying model would likely say that every single sort of the. During that period of time, there's not really enough time, not really enough time has elapsed for there to be sort of meaningful changes in kind of the response rate. And so if your assumed functional form is correct, you should kind of be able to trace out that functional form in that short window of time. On the other hand, if the sort of nonlinear, if nonlinearities are very important, then you should be able to sort of test whether or not sort of the nonlinear. You should actually be able to kind of identify the correct nonlinear form form from that same exact, from that same exact test. And you can repeat this test if you want at kind of sort of periods of time that are far apart to really think about does the sort of shape of the response function change? Now there's a more sophisticated version of this which is leveraging this idea that under intervention these models give different, different predictions. And so what we can actually do is we can think about what would be sort of the spending level that would cause these two models to give as different predictions as possible. And we can then go and we can set for one month, we're going to set spending on this channel to be that, right? It's going to be different than what you're spending now. It might be less, might be more. But what we can then see is we can see at the end of the day which model was sort of closer in terms of getting at sort of what we actually observed in the data. This is what we call this sort of maximal separation test. And it's sort of the most efficient way that you could really think about testing for model identification. But I think, you know, those points aside, again, it's sort of, these points are a little, little technical and are, you know, I realize our goal is kind of to, to, you know, think about these both from a technical and a sort of non technical sort of who cares perspective. And I think from the non technical sort of who cares perspective, you know, why should I care? Maybe that's a better way of saying it. What I think is kind of the main thing is that being careful with how we validate our models and also leveraging experimentation in that process can help us build better models. And it should be the way that we think about doing these kinds of validation. And the paper, you know, gives the details of these two types of tests that you could think about doing to sort of pin down model form. But I think in some sense I don't view those very, the details, the technicalities of those tests as being the main. So what the main. So what is sort of model specification matters because it determines optimal spending policies. And when you're validating models, you need to be thinking about this and potentially doing experimentation that lets you actually validate this particular point. Point.

I love this. First of all, that was an amazing explanation and it really resonates with a lot of the things that I find myself saying a lot and part, I mean, your approach is logically, incredibly sound, very sophisticated. But like, maybe a takeaway from this is just if you're in this world where you're thinking about using an mmm is you should be experimenting more. And the more that you're experimenting, the better. And they don't even necessarily need to be like structured randomized control trials or match market tests. Even just changing what you're doing a little bit and then comparing the model's prediction with what actually happens can be incredibly informative to at least giving you evidence. Like, is the model working at all? And I think this is a thing that I, you know, in the industry I just see people don't do it as much as you would hope that they would. Where it's like the model says, this channel is good and this channel is bad. Like if you go and do something about it, you should be able to see if the model's predictions come true. And that should be a really obvious and core part of validating these models as opposed to a lot of the statistical work that you can do that is often very like prone to error. The just make a change and see if the predictions line up with what actually happened consistently works. And so it's actually like that should be the first place that you go for thinking about doing model validation. And that should be a huge part of actually using an mmm as opposed to a lot of the other more academic stuff, which I think is important and is good, but is like, you don't have to think about it as much when you're thinking about, okay, let's just make some changes and see if it logically lines up with what the model would have predicted.

I think the key, the key point too in some of these things is sort of like these sort of, again I'll use the word like bump up tests or these sort of incrementality tests where you're just kind of taking a channel and you're increasing the spending on it or decreasing the spending on it. The kind of key idea is like, compared to like a true randomized control trial or like a, you know, like a GEO test where you're kind of different markets, maybe you're spending different things in different markets or what have you. The challenge of course, is that, you know, if there's something else that happens to coincide with the time that you bump that spending up, then it can make it really difficult to say, okay, was the increase in spending truly sort of responsible for the increase in revenue or was it the same other thing like Christmas happened, hey, and people bought a bunch of stuff. You know the thing though is that if you're regularly doing these sorts of things it makes it much less likely that there's going to be some kind of random one off event that sort of is going to be completely sort of confounding the estimate of this, of this particular channel's marginal effectiveness. And so I think that this is sort of an important thing. It's not, it's not just about having some variation in X. I mean to fit any model you're going to need variation in X. It's about having sort of some consistent pattern of experimentation broadly defined where X is varying. And it's not varying just based on like oh, we know this is going to be an important season for us, let's spend more. In fact, that's actually the worst kind of variation you can have is it's going to artificially inflate your effectiveness estimates and exactly not tell you what really is the true value of the, of each dollar you're spending on marketing.

I love it. I think it's super well said. I think this is a. You know what I think we both would love to see more in the world is just more exogenous variation, not just as statisticians. I truly believe that that will help more businesses make more money by sort of branching out from what business as usual is. That's how you do the explore, exploit, iteration towards overall more profitability and that it helps the model as well I think is a, is a nice bonus.

On top.

Ryan, we are out of time. This has been an amazing conversation. You are really, really good at this. Thank you so much for joining me and for explaining all of these really in depth topics in a way that I think our listeners are going to be able to understand and really appreciate. So thank you so so much for, for this session today. I really, really appreciate it.

Well, it's my pleasure. I love talking about these things and it's been, it's been a lot of fun. Thanks for having me sat.