Why most MMMs are “probably wrong” (and what to do instead) | William Grosso | Coffee Breaks

The vast majority of MMM usage that I see in the wild, people don't, you know, run randomized trials. Everyone sort of looks at those and says, "Do I really need to do that?" Right? And that's an interesting question. In a perfect world where I had infinite resources, absolutely, I should be validating everything. In a world where I have to make resource trade-offs, does that meet the bar?

Bill Grosso is a seasoned technology executive and data scientist with deep expertise in AI, predictive analytics, and marketing measurement. An AI researcher at Stanford and mathematician trained at UC Berkeley, he's authored or co-authored over 50 scientific papers, advised leading companies across data and tech, pioneered dynamic pricing in digital entertainment, and currently runs Game Data Pros, where he focuses on revenue optimization in gaming. In today's episode, we touched on why teams still use MMMs without validating them, the operational cost of different measurement methods, where you draw the line on what should be included in a model versus not, going from one-shot to a multi-shot problem so models can learn quicker and actually self-correct, and a whole lot more. Let's dive right in. Bill, thanks for joining us. We're so excited to have you.

Thank you for having me. This is a great, uh, opportunity to talk MMs in public.

Uh, well, I'm— we are going to jump right into it. So, um, you and I, you've written some about this, but I think you and I broadly agree but there's a lot of deep problems with marketing mixed models or broadly just observational causal inference when it comes to measurement and especially in the field of marketing. You've commented that, and I think this was in your 2026 predictions or 2025 predictions, that marketing teams are going to keep using MMs without validating them and that this is sort of a big problem. I guess I'd love your commentary. What's going on here? Why is this a problem? Why do people keep using MMs without validating them? Like, what's the background that's causing us to see this over and over and over again?

Well, I mean, so great question. And, you know, nothing like diving right into the deep end. But, you know, fundamentally, I think that not validating is probably too strong. Not validating via randomized trials. Absolutely. The vast majority of MMM usage that I see in the wild, people don't run randomized trials. I think that Recast has launched a GeoLift product. I know Measured has one, et cetera. But everyone sort of looks at those and says, do I really need to do that? And that's an interesting question. I mean, that moves it from, in a perfect world where I had infinite resources, Absolutely, I should be validating everything. In a world where I have to make resource trade-offs, does that meet the bar? And that's a really interesting question. And I think it's, you know, when, you know, the articles I've written on validation, I've talked a lot about how your model is potentially wrong, right? I think in the ad exchanger one or in the mobile dev memo one, Julian and I said, potentially misinforms you, which is a polite way of saying wrong, right? Um, and you know, that's an interesting thing to say out loud, and I don't think we say it enough, and I don't think we say it in stark enough terms, right?

And I would go even further. You said potentially wrong. I think that understates the case. I think probably wrong is like the right way to think about this problem. Observational causal inference, like, you get no guarantees just from running data through a regression.

Like, oh, absolutely none. But, you know, I don't think that people are actually just taking the model and using it, right? In the typical, there's a hierarchy of tests that ultimately wind up at RCTs, ultimately wind up at, you know, GeoLift, etc., etc. But, you know, first and foremost, you know, I build my first MMM as a manager, or I cause to have built, and then I look at the output. And if it tells me something that, you know, all of my smart humans say is stupid, I am extremely skeptical, right? So there is that first-pass filter of, I have experts, they know my business, they've been doing my advertising strategy, they almost certainly live the Wanamaker quote, you know, 50% of my ad spend is useless, I just don't know which 50%, but they have some ideas. And so there's that first pass of validation. There's a second pass of validation that frequently happens, which is, you know, you change your spend and you notice a change. It's not a geo-isolated change. It's not an on-off change. But in general, you do know that you have increased your spend on Meta or you have decreased your spend on Snapchat or something like that. And you do see an output. And so there's that. There's a third pass of validation, which is at least in the field I'm in, which is digital entertainment, right? I don't do e-commerce very much. I don't do— I did consumer packaged goods for a while, which has its own set of issues, which were fun. But now I have an MMP in the mix, right?

So I have the entire last-click attribution Really quickly, what is an MMP for our listeners who are not in this particular space?

So MMP means mobile measurement partner. And so the idea is that if you are primarily in mobile gaming or in digital entertainment expressed on mobile devices, there's an entire set of advertising channels for which you get clicks, right? You get click-throughs, you get generated installs, things like that. And so there's an entire infrastructure which, you know, the first parties, Apple and Google and so on, have facilitated, but which has a small number of these companies, mobile measurement partners, companies like Adjust and AppsFlyer and Singular and Kochava and so on. And their job is essentially to tell you more about where your users came from. And they're not doing it in the MMM sense. They don't do sort of, you know, well, you spent this money on top of funnel ads and it had this downstream impact. There's no multi-touch attribution. It's all last click and it is click, right? But you have that source of truth as well. And so you wind up saying, okay, I've got an MMM. Question 1: Did I do the MMM right? You know, even before you get to the experimentation part. Question 2, does the output of the MMM meet these various sniff tests? Does it make sense with respect to what my smart humans think? Does it make sense with respect to the known variations in spend that I have undergone over the past 6 months? Does it make sense with respect to my mobile measurement partners and the last-click attribution? And so you start sort of, you know, having this suite of evidence. Meta actually calls it the suite of truth, which is— I think that suite of evidence is a better statement because ain't none of them true, but it certainly doesn't sound as good as suite of truth, right? And you start to build this thing around that. And then you say, okay, now that I've got that, now that I'm confident that my model is reasonably well specified, you know, the sampler's happy, and I've got sort of broad strokes agreement on these various things, now maybe I, maybe I do invest in the RCTs. Maybe, maybe I don't. But, and that's a place where I think the industry hasn't really made the case in a compelling way, to be honest.

Wait, let's, let's talk more about that because I feel like I, at least in the sphere that I run in, like the LinkedIn bubble that I am in, people are constantly talking about lift testing. Like this idea of incrementality seems to really be taking off. Are you not seeing that, or where do you think the line— actually, I think I talk to a lot of people who I think rightly believe that you shouldn't go straight to an MMM. If you start with some sort of digital tracking tool, you start with that, and then you add testing, some sort of experimentation, whether it's quasi-experiments or controlled experiments. And only after you've done that do you add the MMM. Are you seeing something different than that, or what is the path that you are seeing where people seem more resistant to the idea of, like, explicit experimentation with, ideally, with controlled trials?

Well, so it's not resistance. Again, you know, it's sort of a squishy notion, right? How much energy am I going to put into it? Am I going to agree with my data scientists who say I should do RCTs? Absolutely. Am I going to prioritize a rigorous and robust system of RCTs that go across all the geos Probably not. You know, maybe I'm going to do one or two here or there to sort of validate that the model is correct in the places where I've checked it, right? But I'm not going to sort of systematically do it across everything, or usually I'm not if I'm a head of marketing. And, you know, you have to explain to me that there's a bang for the buck. So like, I think a lot of Recast customers, for example, are in e-commerce. I don't know that, you know, we don't have a business relationship, but asking, you know, and if I remember the Recast technical documentation correctly, you guys use Gaussian processes pretty heavily, especially to assess the value of channels, right? Okay, so e-commerce, 80% of the spend is in 2 months, right? Happy Christmas. You know, it's a fast-moving space. You know, there's a lot of change year to year. And now I'm using Gaussian processes on this, right? So, okay, am I going to run a national— nationwide experiment or a global experiment? No, I'm not. Christmas is too important. Would I give you St. Louis? Maybe. But now you as a data science team have to say, and that result from the St. Louis metropolitan area generalizes in a way that's valid. It's valid to the entire United States, and the result comes in fast enough to impact this holiday season, and I will be able to reuse it next year. And that level of generalization is a really, really hard bar. It's a really, really high bar, and you know, you combine that with experiments are hard to run and I have limited resources, all of a sudden it becomes an in theory I should do it, in practice maybe I won't. Yeah. And so now you go back to your LinkedIn bubble and absolutely, you know, there's a lot of data science influencers, there's a lot of MMM people who are like, you should do this thing. But there's the counterweighting thing. There's the resource constraints, there's the practicalities. And if that advocacy weren't there, no one would do it. But the fact that that advocacy means some people do it, not everybody.

I mean, we see the full range. And I'll say like Recast customer base, when we first started, we had more e-commerce customers. Today we're actually like not— that is a small minority of our business. We work with a lot of like fintechs and sort of like banks, right? Um, but broadly, I, I agree with everything you're saying. And I think like the, the operational cost of any sort of measurement is relevant when in talking about like what is the ideal path forward for any business, right? Any sort of measurement is going to come with some sort of cost. You know, the costs are relatively low for like setting up an MMP. You're going to have that for other reasons, probably. Um, experiments are costly. MMM is also very costly depending on how you're doing it. I guess the question to you is, I feel like we've been talking a lot in descriptive terms about what do people do, what have you observed, but what is your belief about what people should be doing? What is the ideal state that some marketing team, and again, whether this is in mobile gaming or mobile entertainment or other things, I'm curious, what is your recommendation for what this should look like if you're a CMO at one of these companies?

And, you know, And I apologize for not having a, this is what you should do. You know, one of my customers whom I love dearly likes to talk in terms of how do we get to the Ferrari? And that's a really great way of talking about it. You know, maybe a Ferrari is not the right metaphor, but, you know, it sort of implies there's a range of things, right? You can start with a Honda or Yeah, probably the Honda Motor Corporation is now upset, but I'm implying their low end. But you start with like a used—

But very reliable, very low end but reliable.

But you start with like a used Honda and you wind up with a new Ferrari. And the question is, where's the bang for the buck? Right. And so to a very large degree, this depends a lot on how much money you're spending on advertising, right? If you're not spending millions of dollars per month, why are you doing an MMM? That's a really important question, right? If you say something like, okay, I'm gonna do an MMM and I'm gonna do it right, whatever right means, and what that's going to do is that's going to give me a really nice notion of incremental incremental value of additional ad spend. And so then what I'm going to do is I'm going to put that into my forecasting engine and I'm going to adjust my ad spend and I'm going to get 20% more, a higher return on my ad spend. My ROAS is going to go up by 20%. Okay, that's certainly possible, right? That's well within the realm of what you can do. But, you know, if you're spending $100,000 a month, then that's worth $20,000 and that ain't worth the effort.

Zero.

Don't do it. And we see this in digital entertainment all the time where like customers are like, when should I do an MMM? And it's like, you should do the data collection you need to do now. You should do the, you know, make sure that you're not like losing information because when you do do an MMM, you need 9 months to a year of data. But the reality is that if you're spending $500,000 a month, you should not be having an MMM. The ROI is not high enough for the level of effort. You know, at that point, trust your MMP and pray.

For what it's worth, I sell MMMs and I totally agree with this advice. This is the same advice that I give to everyone. It's just like, you have to think about if the juice is worth the squeeze.

Exactly. So that's part of it. Then the question is, okay, when you think about building your MMM, there's data quality issues left, right, and center. There's, do you have all the data? Which is a different— it's kind of a data quality issue. But like, back when I was doing consumer packaged goods in 2019 and 2020, I was talking to one of the world's leading CPG companies. And, you know, they were asking me questions like, were there trade promotions when they do like manufacturer's coupons or, you know, buy 5 of our products from our various lines and get $20 in a gift card? Do those work? Right. That was the core question they were asking me. And I was like, well, okay, you know, do you have daily sales numbers? And they were like, no. And I was like, okay, let's talk. What do you have? And they were like, Well, we have when we ship stuff from the warehouse. And I was like, and that happens like 3 months before the sale happens because you got to get the stuff on the Walmart shelf before, you know, so you're giving me like these very loose, noisy, we ship some more stuff in anticipation of a sale numbers. That's not really good. Do you know what your competitors are doing? They're like, well, we have these Nielsen reports on competitive behavior. And I was like, that doesn't tell me whether they ran a sale in response to it. And then, you know, even more fun, their structure, and I won't be too specific, but basically they had retail partners who sold stuff and they had line managers essentially who were responsible for one of their product lines with one of their major, one or more of their major retail partners. And so it was entirely possible that they, and you know, they very much believed in the the management philosophy that General Motors used to embrace of having multiple competing lines to do product segmentation. But what that meant was they could actually run a sale or trade promotion in their high-end stuff at the same time they weren't running a sale in their middle-end stuff. And they would cannibalize the hell out of their middle end. And they didn't have unified reporting over that. And so it was like, this is an impossible problem. You don't know what your competitors are doing. You don't have coordination among your various lines, and you don't actually have data on what's happening at the point of sale. And they were like, yes, we understand all of that, but what can you tell us? And I was like, I can tell you to solve that problem.

I've had exactly this conversation. So things haven't gotten better in the last 5 years. I can tell you that.

Right. So there's that aspect, right? So you shouldn't do it if you're not spending a lot on ads. And even if you're doing it, there are tremendous— make sure your data is good, right? And make sure you have the right structural form of your model.

Let's talk about that, the right structural form. So a lot of the— some of our audiences like data scientists, and I think they'll know what that means, but some of our audiences like marketers and maybe don't. Can you talk like, what does that mean?

Sure. So fundamentally, you know, an MMM is a regression, right? You have some equation and then you're trying to find the values of the coefficients along with, you know, uncertainty intervals around it, carefully avoiding both the frequentist and the Bayesian terminology. Uncertainty intervals. I love it. And, you know, the question is like, you know, and I'll use an example from the Recast documentation again. You guys have a great description of spikes. So a spike is an event in time that has both a forward impact and a backwards impact, right? People behave differently in anticipation of the spike occurring. So people have knowledge that the spike is going to occur, it alters behavior. And the question is, you know, do you need in your computational model the idea of a spike? Well, if you're using Robin or Meridian, it don't got it, right? Period. So you are not going to be able to build a model in Meridian that incorporates the idea of a spike in spite of the fact that we know that people change their behavior around Christmas, to use that example again, right? Spend behavior changes dramatically and you can't capture that with a seasonality term. So that's interesting. So one question is, do you need that? Do you need spikes? Another great example, we talked a little bit earlier about Gaussian processes. So if you're using an off-the-shelf Meridian model, and for the audience, Meridian is a great, great, great intro MMM. Google makes it available for free, the source code's available, it's easy to use, but it doesn't have time-varying coefficients. So a core assumption in Meridian is that consumer reaction to channel spend is the same whether it's January or July, and whether it's 2024 or 2025. You're fitting sort of the best possible coefficients given that constraint. That's patently false in a large variety of domains. I do a bunch of work in educational software, and start of school year is just different, period. Right? And that actually goes back to the question about experimentation. I'll I'll return to that in a moment. And in e-commerce, right, holiday season is different. And so you start saying, okay, great. Am I in a domain that has something analogous to a spike? Am I in a domain where I need time-varying coefficients? And if so, what do those look like? And it can get more and more complicated, right? There's an idea of an inter— so a lot of times when you fit a model and you're just, you know, you're starting out at MMM, you'll have your ad channels and their spend, and you'll have a set of controls or predictors, right? And in the very basic form, structural forms of the models, the predictors impact the baseline, which is an interesting thing to do because, you know, a lot of times the predictor actually changes consumers' propensity to react to ads. As opposed to changing the organics.

So an example of this is like, you know, if you're controlling for inflation or the economy, right? That doesn't only impact the baseline. It also impacts the effectiveness of your advertising, right? That's what you're describing.

Absolutely. And that's absolutely the case. Or, you know, I mean, there are cultural events as well, right? You know, more people bet around the Super Bowl, right? So that's interesting. More people are going to download a fantasy sports app around the Super Bowl. People are going to be more likely to react to that advertising. It's going to change the organics, but it's also going to change the efficiency of the advertising spend. And so that's interesting. The other aspect of it is we're talking about the efficiency of the advertising spend. Well, if an ad costs $2 in August but $12 in December because the Budweiser Clydesdales are, you know, consuming all the channels, then that's an efficiency question as well, right? My dollar has— my dollar— my ad dollar's purchasing power has changed dramatically. And so you have to, you know, ask yourself, Do I have that? You know, does my data show evidence that I need to have time-varying coefficients on my ad channels? Does my data show that there are events in the world that impact not just organics but also implement, impact ad spend or ad channels? And if it does, then I need interaction terms as well. And that's another thing that's not in, you know, the standard open source models. And so you start to say, can I even be successful with a standard open source model? Okay, well, now I have to figure out what success means.

And that was— you teed up the question that I wanted to ask because— so you're sort of laying out all of these points, right? And I think I agree with you that these are all things that you need to think about if you're doing the model, but you could sort of lay out infinite number of these, right? Like, we get into this problem where at some point we have to draw the line and say, look, the model can't represent all of the details of reality. That's what makes it— that's how you define a model to some extent. And so where do you draw the line? How do you know if you've gotten to a point where we've made the right assumptions and we've cut the right corners in terms of not having to represent all of reality in this model? What is the success criteria when you're doing this?

So first and foremost, let's pretend you did some of that analysis and you said, You know, it just looks like I need interaction terms and time-varying parameters. Those are fairly common decisions, right? Those are not like anybody's going to stop and say, whoa, that's an obscure thing. So I've got that. Now, how accurate is my model, right? And that's a place where we as an industry have been really bad as well. So the standard thing that we do as an industry is we talk about MAPE. Basically prediction error, right? And we talk about prediction error as a percentile, but we don't talk as much over what is the evaluation criteria, right? Is this an in-sample test or is this an out-of-sample test? You know, is it MAPE or is it SMAPE, symmetric MAPE, or is it WMAPE? What is the best we can do? Right? That's an interesting question. And we don't really— there isn't like— you can't go out to ChatGPT and say, give me an authoritative reference for what a good 28-day out-of-sample WMAPE in e-commerce is for an MMM. Nobody in the entire industry is willing to say, hi, I know I have an answer to that question and here's the bet I would place. Right?

Yeah.

So that's interesting, right? I was on a call a couple of months ago where somebody said, yeah, you know, a 20% MAPE is pretty good. And I was like, exactly. I did that face.

No, it's not.

But, you know, that's me. And I'm a vendor, right? You know, am I really going to sit there and say, yeah, he's, you know, Yes, I am. I'm doing that now without naming it. But, you know, a 20% MAPE, not even a WMAPE or an SMAPE, is just a very naive criteria. But is the idea of MAPE even a good idea? You know, I think the actual question is going to be forecast error, right? So out-of-sample MAPE, yes, that's a place to start. And then there's a pragmatic criteria, which is— we use out-of-sample WMAPE, to be honest. We don't really— if you're not really spending much, we don't care if we're wrong. Let's focus the accuracy criteria on the channels where you spend money. So now what is a reasonable number? And the answer is sort of twofold. So one is what are you going to do with it? Right. And that's a place where I think the industry does, you know, and this is a place you and I have talked a little bit in email about the evolution of the industry. I love the fact that people are starting to take spend forecasting seriously. And I think that's a great place to also incorporate experimentation, although nobody seems to really do that yet. But you shouldn't have like a budget optimizer that says, you know, if some divine being were to reallocate your spend right now for all eternity, here's the numbers and here's the proportions to what channel. That's not the right way to do it. And it's not the right way to do it because, you know, you're going to increase spend in some channels, you're going to decrease spend in some other channels. You're probably going to go outside of like what is reasonably generalizable. You know, there's this great story that Julian, my coworker, tells about, you know, a company, you know, every time they told the MMM to optimize, it would recommend spending money in the high-risk channels because those were also the high-reward channels, but those were also the channels where they had the least data. Yep. It's a reason those channels were high-risk, and it's because all the experienced humans were like, I don't really want to put a lot of money in that. Right? And yeah, a naive budget optimizer looks at it, takes the point estimate of the value of the channel and says, pump more there. And the answer is no, just no. Yep. Create a spend flight with a weekly spend plan change, you know, take into account the fact that some of these are strategic spends and aren't going to change. And then maybe you have a budget optimizer target and you work towards it and you gain confidence along the way. But you're also, because you're now saying, I'm going to do weekly spend plan changes and I'm going to have weekly variations in spend, heading towards the optimizer. I also get weekly checkpoints. Was my model correct or close to correct? And I also get the chance to sort of naturally, without doing an all-or-nothing geo test, nonetheless vary spend in ways that tell me incrementality.

Yep.

And it sort of naturally becomes part of the forecaster as opposed to being a separate product or a separate set of work.

So I think So I love everything that you're saying. I'm going to sort of restate this in a slightly different way because I think it's a really important insight for the industry. Because I think when you think about legacy or the traditional approach to media mix modeling, it was very academic approach of like, we are going to assemble a team, we are going to collect the data, we're going to then spend 2 months cleaning the data and 3 months analyzing the data, and then we are going to publish a report. And that's the goal. The goal is to publish a report that has some insights into it, in it where you ostensibly learn something. But I think that is a very academic approach, right? Academic economists are very comfortable with that world. But the problem with that approach is that it's very difficult to verify or validate, right? It's very difficult to know if you got the correct results. But what you're describing is actually really changing the problem and trying to change it from a one-shot problem where it's like, oh, we have to get all of the economists in the room to build the best possible model so that the single report is correct. If you're in that world, that's very hard to do, and it's hard to ever know if you got it right. But if you can change the problem to be a multi-shot problem where you can do explore-exploit, all of a sudden the problem gets easier in some ways in that there's a lot more engineering to do, But you don't have to be like, oh, we have to invest all of this time to make sure that we get the one model right. Instead, if you can build a system that's a little bit dumber on any individual task, but that gets smarter over time and can learn more rapidly, you can do much, much better. It's sort of this, it's like, it's not obviously like a bandit-style algorithm, but it's the same idea of actually, can we make a system that maybe any individual says, decision isn't perfect, but if it learns over time and it can learn quickly, we can actually outperform these other more advanced methods that take a lot more time, like that just aren't as responsive. Does that make sense? Does that resonate with you?

That totally makes sense, and that's a big part of it. Um, there, I would decompose it as, you know, the, the academic approach, you wind up with a model, and maybe it is even really a good model But it's a good model for that point in time. And then the world changed, you know, and the deep truth is the end of the world changed happens every day. And so, you know, taking that approach and having a system which doesn't adapt is more or less useless, right? Not entirely useless, you know, in their scenarios, but For the worlds I live in and the places I do my work in, things change almost constantly. And, you know, I mean, a great example in digital media is the rise of AppLovin, right? This is now an amazing force in digital advertising, especially in mobile gaming, especially in digital entertainment. If you had built your MMM 2 years ago and were relying on it for insights, that's an interesting mistake. Right?

Yeah, it's like Apple didn't even have its advertising product 2 years ago. It's like totally, it's so totally new.

So you stare at things and you're like, given that level of change, you know, you sit there in digital entertainment and you're like, so in 2019, Apple decided to blow up the mobile advertising ecosystem with IDFA deprecation. And Wow. The number of changes to the infrastructure, the number of changes to everything, right? What measurements are even possible has changed dramatically. And so you wind up in a place where, you know, it's a lot more engineering effort and it's a lot more human effort too, because like if you're running a major entertainment property, you probably have, you know, on the order of 500 to 1,000 distinct channels you're tracking. And there's no way to build an MMM on 500 channels, at least as far as I know. So you aggregate them in various clever ways to sort of get core features. And then you start saying, okay, now that I have this model and now that it's making predictions and now that it's advising me and this is my 1 to 2 month spend plan, I need to go and change all 500 channels. Right? And that's interesting as well. And so the thing to the core parts are the world changes enough that you need to learn. Models don't generalize. You're going to go outside of historical norms in some spend channels, and that means your saturation curves are open to question, right? They're just— who knows, right? And that's also an interesting thing that we as an industry don't talk about. Like, okay, if you're using PyMC, you have You have hyperbolic tangent, you have logistic saturation, and I think you have Hill, right? Hill function. Okay, great. Which one should you use? It's a really interesting question because the whole point of an M— or one of the big points of an MMM is it gives you incremental ROAS, right? It says you put another dollar in, you're going to get $1.17 out. In order to do that, you need to have the saturation curve right. Okay, how do we validate the saturation curve part of things? That's a really interesting question, right? But that's another place where if you build a spend forecaster that says weekly flights and changes on a weekly basis, you can sketch it out and sort of gain confidence. Maybe you don't have the right, uh, parameterization of your logistic saturation, but you can at least validate that it's logistic saturations, which is high value.

Yeah. Or you can check that it's working at all, right? I always have more fundamental doubts. Like, if you're in the world where it's just like it's one of these two saturation curves, that's an amazing place to be at. You've solved most of the other hard problems. Most people are in the world where it's like, could this even possibly be correct? But again, I think the thing that you're describing is so aligned with our vision on the right way to solve the problem. Is not to try to build the perfect model. That is going to be impossible. And even if you did, it goes out of date very quickly. What you need is you need a system that can sort of self-correct and helps you learn over time in a smart way. And if you can do that, if you can, like, if your random walk is actually going up, even if some weeks you lose and some weeks you win, but on average you're going up, that can yield a huge amount of improvement over 3 or 6 months. If you can do it in a, in a smart way. And so I think that is so much— it's so much better to invest your time in doing that, in like the learning process, than it is in being like, oh, we're going to hire 7 PhDs to build us the perfect model. Because again, that's just like so limited in terms of how quickly you need to be able to respond and adjust over time.

Absolutely. But now let's take it back to the question about RCTs, right? So what we've just articulated is that we both believe that a good enough model is better than a perfect model, because— and a good enough model that is capable of adapting and learning over time is better than a perfect model, better than a perfect one-shot model. Great. Now, when you say something like, I'm going to turn off spend in this metropolitan area for these channels in order to assess and run a geo-targeted experiment, Is that really a bang for the buck moment? And that's an interesting question. And if you're spending $500 million a year in advertising, yes, it is. But you have to be pretty far along that, you know, pretty far along the path to $500 million for that to be the right use of resources. You know, I would argue that getting spend forecasting right, getting spend flights right with varying on a weekly basis and sketching out your saturation curves and verifying that your 28-day forecasts are accurate are far more important than anything else. And that's actually a lot of work too, right? And it's like— and another thing that happens, like, you're one of these companies that experiences hypergrowth. Well, hypergrowth is fun, right? It means you found something that's working. Yay. It also means that your advertising strategy is probably like quadruple the spend in all channels, right? Multicollinearity jumps up and said, yeah, it's like, oh, it turns out we're working. We have product market fit, 4 fuel on the fire, double spend in Meta, Snapchat, and TikTok tomorrow. Okay, but if I do that that way, I have a hard— any model, any regression model is a really hard time disambiguating what's due to what, right?

Yeah.

So inside your spend planner, you should be saying not only like, okay, I've got a budget optimizer. It says, hey, this is the amount we should spend in these channels and we're wrong. That's interesting. And okay, but some of those are in channels with a fairly wide credible interval. So let's, you know, be cautious and do it incrementally. And some of those are going outside the boundaries of historical spend. And we know that we're often not entirely accurate when we go outside of a generalizable zone. So let's be cautious there. And also, this is an opportunity to increase Meta by 20% and decrease Snapchat by 20% and break the multicollinearity. Right? And that's something that— and maybe yours does it, I don't know, but I don't think any commercially available spend planner really stops and says, there's an opportunity when constructing a spend flight to break multicollinearity and make the model more accurate.

Yeah, so this is such a great point. Um, and I think— and I'll just flag, like, you, you, you talked about the hypergrowth scenario. That's definitely one scenario where you see this. But even just normal marketers, right, they like to think in campaigns, especially like brand style marketers, right? They think in campaigns. They're like, look, we're going to do the X campaign, and for the X campaign, we want to do radio and and podcast and TV, and for it to have the most impact, it should all go out at once. And so they do that. Like, that is how they plan their buys, and it generates this, this problem. So in general, like, the default for marketing, again, especially in larger organizations where they think in— like, they think more like brand marketers— like, the default is multicolinear, and they do it on purpose, and they do it like with intention, because that is how they think about designing their campaigns to, to bring some message to an audience. So we see that a ton, and you're totally right, there's often opportunities to like learn more by doing things slightly different. And I think we actually— at Recast, we have a tool that does this. It's in beta and it's like definitely a little rough around the edges, but it tries to simulate like, okay, what would an optimal flighting of this be where the goal is to maximize learnings, right, as opposed to maximize revenue? And so the ways that you can start thinking about it. But I think there's also this like cultural problem of you need to get marketers to think about learning as a good, right? Like as part of your campaign, you should think about, yes, maximizing impressions and maximizing return, right? Whatever those things are that, that you marketer think about. But we need to at least have like learnings somewhere on the list. And it— there will be times when maximizing learnings or even getting learnings at all isn't the priority. And so we put it to the side, but it should be part of the conversation or part of the question as we are thinking about planning these campaigns over the next 6 to 12 months. But that's like a cultural thing and it's not just a marketing question, it's like an organizational question of what do we prioritize? What do we care about as a business?

No, it's also a question of how easy is it to do.

That's also true.

You know, so we've actually done wireframes of a tool we call Scenario Man for Scenario Manager, right? And, you know, I mean, I constantly say this, but, you know, nothing I do will like ever rank me up there with Newton and Einstein. You know, we copied ideas like when you first start to become a little affluent, You'll get these emails that say, you know, your risk profile in your E-Trade account is out of whack with where it should be given your age, right? Like, and it's like they talk about in terms of like aggressive, moderate, conservative, right? Okay, well, let's— what if we actually had like a dial like that on your spend plan as well? Are you being aggressive? Are you going for the highest expected value? Are you going conservative, going to decrease the variance? Those are actually two very different spend plans a lot of the time. But explicitly having someone say, you know, right now I want to be aggressive and then altering the spend plan, that's one thing. We explicitly inserted a step in the flow that's like, okay, now that you have your spend plan, would you— how— what percentage of your spend would you be willing to devote to having a better model next month? That's, that's the question, right? It's not like, would you change this, would you change that? It's like, in general, would you spend 5%— would you alter your 5%— would you be willing to take a 2.5% revenue hit reallocating 5% of your spend? Yep. Because at that point, it's an easy yes. I don't have to think about it and I know what I'm buying. I'm buying a better model next month. And that's not something that we ever do as data scientists, right? You know, and that's an interesting thing. Even like in this conversation, I said you can break multicollinearity and you were like, yes. And it happens in campaigns as well. But, you know, as soon as you go to the C-suite and you say, we've got a problem and it's called multicollinearity. It's like multi-collinearity. Too many syllables. You know, we can get a better plan next month if we invest some in learning this month is a statement everyone understands. Yeah.

And I think what we've found is useful, because again, like talking about multicollinearity, it's such an abstract problem. It's always unclear, like how much it matters and you can't ever really get rid of it. It's, it's just like, it's a very hard concept and not, not, I just don't think it's a very fruitful concept to think about. What we talk about is we talk about, it's like we try to bring everything back to the forecast. We say, look, next month's forecast, there is, we have a problem that generates uncertainty in next month's forecast, right? There's uncertainty in next month's forecast. We can reduce that uncertainty if we can learn more about what truly works.

Exactly.

And we can learn more about what truly works by adding more variation in the spend, and that will improve next month's forecast. And so really, like, yes, like, I know that it's multicollinearity that's a problem here, or it's like, it's, it's, there's not enough signal, but the value to the business is we can make next month's forecast more certain. And that is the thing to really focus on. Exactly. We can reduce risk. And that's a thing that people are actually willing to spend a lot of money for.

Exactly. So now you say something like, we have a budget optimizer in the Robin Meridian sense, um, and it gives me in some sense the ideal channel allocation based on point estimates. But it's got a really wide confidence interval problem here, or credibility interval. You know, it's got a lot of uncertainty.

Uncertainty interval.

Yes, right. You know, great. Maybe you have to live with that this holiday season because you're out of time. But how do we fix that? What are you willing to spend for, you know, to reduce the uncertainty? And that's a conversation you can have. And that's a conversation, you know, switching gears a little bit. So we showed the scenario manager wireframes to a friend of mine who's a CFO, and he literally said, I want this. And I was like, you can't have it. It's Figma.

You can have this one. It can't have your data in it, but you can have the screenshot.

Yeah. But his reaction was literally like, I've been waiting my entire life to be able to have conversations about marketing spend in terms of risk and financial forecasts with my CMO. And we have those conversations, but we don't have any tools that support it. And so it winds up being a very hand-wavy, you know, maybe there's some Excel flying back and forth sort of conversation. And I was like, really? Okay. But, you know, that's, that's the thing. And so a lot of what we need to do as an industry, and again, going back to the experimentation question, is the problem the RCTs solve is they reduce uncertainty. Great. They help us believe that the models are accurate. Great. But they're not the number one tool in either case. There are better ways to reduce uncertainty. And there are better ways to validate model accuracy, at least at the low and middle ends. And so that's sort of like an interesting thing. That's not to say, you know, when I say like, when we say that observational causal models are often wrong, they are. It's just immediately jumping to the RCT is usually a mistake.

Interesting. Yeah. I agree with that. I'm glad we've closed the loop about 40 minutes later.

So we've got— conversation though.

This has been— no, this has been great. There's— so we only have a couple minutes left, um, and there's one last point that I, that I wanted to hit on. So you wrote a, a postmortem on scientific revenue that I found really, really compelling, um, and one particular point that I think aligns with what we've just been talking about here is, is you made this point that you can't optimize from the outside, um, and at least the way that I interpreted that is that there, there's some degree to which companies don't want automation, right? And so we were talking a lot about these systems and about like, oh, you know, you need to make things easier, but you still, I think, need to leave some decisions in the hands of, you know, the CFO, the marketing team, how they're thinking about doing the planning. And I guess based on your experience, like, how do you think about that? Like, where is, where is that line? How much automation can you do for people? How much should you do for people? What needs to live on the side of the business? I'd love for you to talk a little about more about that, especially in the context of experimentation and, you know, spend forecasting and planning that we've been talking about here.

Yeah, no, this is a big and meaty topic to go into in the last couple of minutes. You know, at a high level, the point I was trying to make was slightly different than that. You know, so as a vendor, as a consultant, as someone who helps companies, right? You never get told everything. There's always information that's just not shared with you, and it's usually a surprisingly high percentage of knowledge, right? There are databases you don't have access to because there's PII in them and no one's ever really put the access controls in place. There are significant corporate events that weren't fully shared even within the company, much less to you. There's all sorts of things like that. And so, what you wind up is you can build tools for people, you can build automated tools for people, as long as your definition of automation is pops up a dialog box, you know, doesn't automatically make the change, but says, "Hey, there's a problem here," right? You can do alerting and you can build, you know, so you can build tool change, you can build things like M&Ms, You can, you know, help them understand the structural forms, but there's sort of two overriding facts. One is you're eventually going to go away, and two is you're not going to have the full picture. And both of those argue that, you know, closing the loop needs to be done on the inside. The place where we try and define that boundary in our practice is we, you know, for MMMs at least, is We will, you know, either review what you have and give you suggestions on how to get better, or we'll build you a prototype of what we think you need. To do that, we'll do some initial EDA and we'll say, hey, you know what, you know, evidence is that you do need Gaussian processes and things like that. We'll sort of like talk about, you know, don't use MAPE, use WMAPE. But when it comes to actually sort of deciding on the final structural form of your model, and doing the EDA on what data parameters are necessary, what controls should you have, we actually believe that should be done internally because it's a decision that you're going to change in 3 to 6 months. The business context will change. You'll have new information and you need experts on the inside that know how to do that. And that's one of the, you know, and that's sort of the point is If I'm on the outside and I'm in charge of that, of figuring out what controls you have, then you need to call me up every month as the business conditions change. And you need to share information at a level that you're probably— it's not even that you're uncomfortable doing so, it's that you're not built to do so. Yep. Right. And so if you say that the external party is going to be the automation and optimization person, you're, you're, you're just signing up for failure on a 6-month horizon. You know, there can be superhuman efforts that get you initial results, but then it's still going to fall apart.

So I think this comes— I'll try to close the loop here. This, it comes back to this idea of really when you're thinking about doing this, especially in the world of like measurement and forecasting, is if, if you're an organization, you're a brand, you need to be thinking about how are you building the system. Right? Don't treat it like it's a one-shot problem. Treat it like this is a system that you are going to need to use for the long run, and that is what you need to be focused on building. And if you can build that and think about, again, what do you need to own in-house in order to support this system for the long run? And that is what can help you make decisions here in terms of doing the right thing. If you treat it as a one-shot, it's going to go out of date. The outside vendor isn't going to have enough context. You need to think about how do you build the system for the long term.

Right. And so one of the things we do is we'll also give people staffing plans, you know, for the initial build-out, right? You know, building a house, you need a team of carpenters. Afterwards, you need a handyman and a plumber and electrician and a guy who mows your lawn. Different scenario. You know, but you need staffing and they need to be expert in it. One of the things we always say is, you know, You know, maintaining a good MMM isn't nearly as hard as initially building it, but you should still have two data scientists. And people are like, two? And it's like, well, so here's the thing. What happens if one of them wins the lottery? If you only have one and it goes away, you've got a problem. And the vast majority of my ideas are pretty dumb. But you know what? When I say them out loud and someone says, "Why do you believe that?" They go away. All right? So having a sounding board, high value. Having redundancy, high value. If the business context really, really changes and all of a sudden there's enough work for the next 2 months for 2 data scientists, having those people, high value. So your staffing plan for an MMM is probably 2 data scientists, neither of whom is full-time on the MMM. Okay. But that's something you need to anticipate and plan for regardless.

Absolutely. Bill, we're out of time. The hour flew by. This has been such a wonderful conversation. We're going to wrap here. Thank you so much. Thank you so much for joining us. This is just like a really great sort of in-depth conversation about all of, I think, the really interesting stuff that's changing in the world of marketing measurement, MMM, and experimentation these days.

Yeah, no, this was a lot of fun. Thanks for having me.

Thanks, Bill.