Validating your MMM with Recast Co-Founder Michael Kaminsky

All right, welcome everyone to the Measureup podcast. Today on the show, we're going to talk about the fascinating, nay titillating topic of marketing mix modeling and specifically how to validate the results of your mmm, your marketing mix model. Now, before you hit skip on your podcast player, I promise you will walk away from this episode with a greater understanding of evaluating your model and an ability to know whether it's bullshitting you. Luckily for you, dear listener and me, I'm joined by an expert on this topic. Michael Kaminsky is the co founder of Recast, a marketing measurement platform that helps brands eliminate their wasted marketing spend using marketing mix modeling. Welcome, Michael.

Thank you so much. I am so excited to be here. Thanks for having me. I am excited to get into this titillating topic.

All right, I hope we didn't oversell it too much, but I think, I think we'll deliver on the results here. But you know, before we dive right into this, this core topic of validating MMM results, can you let us know a little bit about how you got to where you are today? What was your journey to starting Recast and tackling this complex and age old problem of marketing measurement?

Absolutely. So I'm an econometrician by training. I started out on sort of the academic side of statistics, doing a lot of causal inference on observational data in a bunch of different domains. Did a bunch of work in healthcare, did a bunch of work on impacts from environmental changes, a bunch of really interesting, fascinating work. And I always wanted to find a way to do that in industry but without having to go into academia. So I moved from that academic world to really focusing on marketing science. I ran the data science team at Harry's, the men's grooming brand, selling razors online and now everywhere. And from there I got a lot of exposure to all of the really hard problems in modern marketing, which was was really fascinating. Spent a lot of time thinking about how can we use data to make better decisions, specifically in the marketing domain. And through my experience there, I got exposed to this idea of marketing mixed modeling, this idea of doing causal inferential statistics on very complex data in a very complex world. And it really sparked my interest. I got really excited about it and I thought that the way most people were approaching the problem wasn't quite right. So I started Recast to really try to tackle the problem head on. It ended up being way more difficult than I'd even imagined at the time. Much longer journey than expected. But it's been really fascinating and a lot of fun to really get into the weeds of how does marketing work in the real world and how can we use modern statistical tools to help us make better decisions. And that's really what it's all about.

Yeah. You mentioned it took longer than you thought. I think I've heard on maybe previous podcasts you were on that, you know, you thought you would take a few months to sort of build up the model and then start to run from there, and it ended up taking a couple of years.

That's exactly right. Yeah. My co founder Tom and I thought that we could use some off the shelf tools and we would be able to sort of plug in the marketing modeling framework, get good results out and we'd be like off to the races. We thought we were going to build a nice little lifestyle business and then as we got into it, we can use those tools to get results. Right. Getting results is not hard. It's getting results that are accurate. That's the hard part. And that once we sort of figured that out and we were like, oh wait, this is actually going to be much more challenging, we had to really rethink everything, rethink all of our assumptions, rebuild the whole modeling framework, custom from the ground up, no off the shelf tools. And there were definitely points in that multi year long journey where we were like, this is impossible. There's no way to do it. We're going to have to just like hang up our hats, turn it off. Can't be done. Eventually we had a couple of technical breakthroughs that allowed us to get to the model performance that we were really looking for. But it was a really long journey and way, way, way more difficult than either of us had imagined. And it also gave us a lot of confidence that, you know, this is a viable business because most companies aren't going to have the resources to put, you know, multiple statisticians on a research project for multiple years to try to get to the other side of getting a marketing model that maybe works. Right. And so it was exciting for us as well to hit those technical breakthroughs, get to that point where it's like, okay, yes, we have a product that actually works, is exciting and then be able to go take that to market.

Yeah, yeah. I'm not a fan of golf, but like the analogy is that, you know, it's a. Takes a day to learn and a lifetime to master. Right. It's like marketing maybe some similarities there. Right. It's easy to nowadays with, with open source packages like Robin and Lightweight. Mmm. And the like. It's, it's really easy to get up and started with them. But yeah, to, to really go deep, it's, it's a rabbit hole. You go down and you realize that there's so much more to it. So, so many more decisions you have to make and ways that you have to think about it and pitfalls you have to avoid. And before we kind of go too deep into this for anyone listening, that's maybe new to MMM marketing, mixed modeling, what would you say? Like what's the, what's the, you know, the, the one minute explanation of what is this marketing mix modeling thing? What are we doing here?

Marketing mix modeling, the concept is, is not that complicated. The idea is we want a top down statistical model that finds the relationships between marketing spend on one side and then business outcomes on the other. So the idea is to answer questions like when we tend to spend more money on some marketing activity, more money on tv, more money on radio, more money on Facebook, how many additional conversions or additional leads or additional revenue are we driving on the other side? And you can imagine doing a very simple version of this in Excel. You know, some weeks we spend more on some channels and that drives more, relatively more revenue and other weeks we spend more on other channels and that drives relatively less revenue or whatever. But as the channel mix gets more complicated when there's more than one or two channels, that Excel analysis starts to break down, it gets really complicated. And so that's where you need to bring in statistics to really use statistics and, or machine learning to find those relationships and then do it in a way that can actually help you again, drive the business forward. We should reallocate dollars from an underperforming channel to an outperforming channel. That will help us get to an overall higher level of efficiency for our marketing mix.

Yeah, and I'm probably somewhat in a bubble because this is where I spend all of my days is looking at marketing mix modeling and marketing measurement. But I feel like this is becoming a topic that is really starting to get a lot more attention as things like privacy regulations and technology changes around things that Apple's doing with ITP and ATT and cookies going away and all of this kind of stuff is happening. I'm seeing a lot more conversations on Twitter, on LinkedIn about. Mmm from, from, you know, every day I see someone new posting on LinkedIn about, here's what, here's the difference between multi touch attribution and marketing mix modeling and why you should.

You really are in the bubble.

It sounds like that maybe it is. I'm Hoping I'm trying to make the bubble bigger so include more people into it. But yeah, I mean, what's your, what's your take? I mean, I think I know your take, but you know, a lot of marketers have been used to multi touch attribution for the past decade and now they're starting to kind of slowly come around touch the idea that maybe that's not working or never has.

Yeah, I think that's exactly right. I mean there was a time when we were, when we could track people across the Internet really well, you know, you look back to like the early 2000s when a lot of the like big e commerce brands of that era were being founded, Harry's being one of them. But like Warby, Parker, Casper, there were a bunch that really blew up in that time. And there was an argument to be made that with digital tracking, multitouch attribution, like you could capture most of the customer journey. A few things were different back then. One is that there were less social media channels, right? There was. Facebook really was like the big one. There wasn't a lot else going on.

Google plus. Don't forget about Google.

Yes. Oh, sorry, I forgot about Google. Yes, a huge success of Google. So there was the Facebook wall garden and then there was Google Search. And those were your two big digital channels. Like maybe you did some display ads, but really like that Facebook and Google Search was it. And so if you're advertising only on those two channels, you're selling only on E commerce online. And with cookies and being able to track, you could see most of what was happening leading up to a purchase. And that could give you some amount of confidence that okay, yes, we can use digital attribution in order to estimate the incrementality of our marketing dollars. Always some questions there around true incrementality. Branded search, we've all heard about those experiments you turn off branded search. Revenue doesn't change. But in general, not that bad. But today's world, totally different, right? Tracking has gotten a lot harder. Rise of ad blockers. Apple's ATT with iOS 14.5 obviously. Plus there's just a lot more channels, right? A lot more digital channels. People advertise on Snapchat and TikTok and all of these other places. And then all of a sudden you really have to start to worry about, well, where are the gaps, right? What if someone saw an ad on Facebook and then saw an ad on TikTok? How do we think about that? What does that mean? Are we going to be able to Track that. And then that starts raising really hard questions for marketers and marketing analysts to answer. And so people have started to think about, okay, let's take a step back. Let's think about what's the problem that we're actually trying to solve? The problem that we're trying to solve is not attribution. Right. The problem that we're trying to solve is incrementality. And solving incrementality. Right. Requires different tools. And so that's where people start to talk a lot more about experimentation, about marketing mix modeling, different ways to get at this attribution question outside of just tracking people across the Internet, which is getting, which has gotten a lot harder over the last five or 10 years and is only going to get worse from here.

Exactly. Yeah. And so, you know, as people start to come into, you know. Mmm. You know, there's a lot of.

A.

Lot of things that people will, you know, call out as being, well, you know, it's, it's top down and you can't do, you know, granular analysis with it and it's, it's slow and it's, you know, you know, you can't use it for this and that. There, there are certainly limitations with, with mmm. What would be some of the, you know, some of the bigger limitations that you see on the marketing mix modeling side?

Yeah, definitely. So biggest limitation is going to be granularity. Marketing mix modeling is a top down statistical model. And that means that your, the modeling is going to be fundamentally limited by the amount of signal that is in the system. And that means that you can't really slice and dice the data the same way that you can with like a digital tracking methodology. So like getting down to the ad set or the creative level is very, very difficult, practically impossible with mmm, you lose a lot of signal there. And so being able to say like, is the red background or the blue background ad on Facebook or Snapchat better? It's like, mmm is not going to be able to help you answer that question, period. And so that is like a really core limitation. Other things that go into it is speed of response. Right. At recast, we've invested a lot of research into being able to make the process of doing an MM much, much faster than what was possible. Historically we're down to, you know, every week instead of every quarter or every six months. But still every week is not every day or every hour. Right. We're not seeing data as it streams in like you can on the Facebook platform or in Google Analytics. And so those Two things are really definitely limitations in some meaningful ways. And those are the things that I think when marketers are making the transition from like only using Google Analytics or only using some digital attribution tool to bringing in mmm, those are the biggest things that they have to get used to is like how do we think about how these different tools operate at different levels and different timescales and how do we bring that together to operationalize it for our business? How do I make decisions on an hour to hour basis so versus a week to week basis and those are going to be different. And I think. But that requires learning new skills, learning new patterns, figuring out new ways of working in order to bring that all together.

Yeah, yeah, there's still, there's, I feel like there's still a lot of confusion or maybe misunderstanding of kind of the current state of mmm. Right. I have a client the other day I was just talking to where, you know, they get from their vendor, you know, it's every six months they get a report and, and it's, you know, by the time they get it, it's already out of date and they're still using it to make decisions around budget and stuff. But that speed, you know, there's still some vendors that are doing it that kind of the old fashioned way with maybe some outdated methodologies. And you know, there are, things are changing quickly. A lot of tools like Recast and others provide these capabilities in a much more frequent type of reporting cadence, which is, you know, as we go through this kind of growth of MMM becoming more important, I think there's a lot of getting through that, you know, transition from old to new and getting people to realize that. Yeah, it's not, you know, it's not the MMM of 1990 that, you know, was only used by major CPG companies. Right. It's something that is a lot more accessible, a lot, a lot more nimble than it used to be. So it's good to.

I think that's right. I mean, I'm really excited to be in this market right now because we're going through so much change. There's a lot of exciting things happen. A lot, a lot of exciting things happening. A lot of changes to the fundamental technology and the way people use it. And like really what we've come to realize at Recast is the product that, you know, the CMO of Pepsi bought 20 years ago for MMM once every six months. It's just, it's almost a completely different product than what we're selling. It's, you know, that's backwards looking. How do I justify what happened in the last six or 12 months versus something that's forward looking, that is much more performance oriented? The like, you know, I'm doing scare quotes for the people who are just listening, but the performance marketing revolution of the last 10 years has actually been hugely impactful. And I think in a lot of ways it's the right way to think about marketing. You know, where is the performance, how do we measure incrementality? And a lot of the MMM methods and big vendors were developed prior to that and so they don't really think about the world that way. And so it's a very exciting time to be here. But a lot of things are changing really rapidly. And so a lot of the assumptions that people have about how MMM works or has to work just aren't right anymore. And we're going to go through, I think, another couple of years of a lot of change, a lot of new products coming out and it's going to be really exciting. And I'm excited to see where we are five years from now. I mean, I believe that Recast is going to be the winner. We're going to have sort of all of the best tools, but definitely a lot of change coming over the next couple of years.

Yeah, absolutely. So, you know, a lot of things changing in mmm, but like we talked about, there are still some of those limitations. One of the limitations we want to talk about today, the main topic is around validating the results of mmm. Like, like you mentioned, it's easy to get results out of a marketing mix model. But how do we know if those results are valid?

This is a really good question and it's, it's the most important question. But it's, it's also a difficult one because it's very, it requires thinking about the world in a very different way than what a lot of people are used to or what they've been exposed to. So the thing about MMM is that it's a causal inference problem. What we want to estimate is incrementality. If I spend an extra thousand dollars in this marketing channel, how much additional revenue or additional leads or additional whatever am I going to generate? And that's a causal inference problem. And causal inference problems are very different than like machine learning type problems, which are prediction problems. Right. With machine learning type problems, you're really just trying to predict some phenomenon. Can we predict revenue? Can we predict something? And it's easy to test those, it's easy to understand their limitations. Causal inference is much harder because in general, we don't have ground truth that we're comparing to. We don't know what the true incrementality of Facebook was yesterday or last week or the day before. And so we're always in this world of uncertainty. And that is really what makes MMM very different from a lot of other modeling environments and a lot of what other people are exposed to. So that's the problem. The problem is, the problem that we're trying to solve is causal inference incrementality of marketing channels. We don't have any ground truth. And so we have to think really hard about how are we actually going to validate these results in order to get to something useful. So there's a bunch of different ways that people sometimes think about validating an MMM that don't really work very well. So we can sort of talk through what those are. So one thing that people often fall back on are simple goodness of fit metrics, right? So a lot of people talk about R squared. And you know, most people took in at least an undergraduate statistics class, they got exposure to this idea of R squared. It's how much variation is explained by the model. In general, R squared is not going to be a very good measure of marketing mix of a marketing mixed model performance.

Oh, but it's so seductive. My r squared is 92%. I've explained everything the problem, and it is seductive, right?

And this is a huge problem for us because we like don't use R squared. But everyone always asks us what the R squared of our model is and we have to go through a whole journey of explaining to them why that's not the right question to ask. So the problem with R squared is most of the time R squared is measured in sample. And so now we're starting to get in the weeds of statistics. But when we talk about in sample goodness of fit metrics, we're talking about goodness of fit. How well does the model fit the data within the data that the model has seen? And so if we're using in sample R squared, the problem with it is that it's really easy to game. If you give me, if you tell me that I should just try to maximize R squared, it is trivially easy for me to build a model that has a 100% R squared, right? That is perfectly fits the data. It's trivially easy to do. And this is a thing that you actually see with a lot of less scrupulous MMM vendors is they go in and they add a, a bunch of dummy variables on every day where their model doesn't fit well, causes the model to fit perfectly and so then they can point to this really high R squared. Unfortunately, the thing that we actually care about, causal inference, doesn't tell us anything about that at all. So it's really like a misdirection and it often leads people to this thing of overfitting. Right. They overfit to the data that the model has seen and that causes them to have worse predictive performance on data in the future. And since we're trying to use this MMM to make decisions about the future, how much should we spend on Facebook versus TV next month in sample doesn't really help us. Right. What we care about is how well can the model predict the future because that's what tells us can we actually use this to make decisions in the future. And so goodness of fit metrics, not really a good way of thinking about validating the model's results because it doesn't tell us about the causal inference problem.

Yeah. And so in terms of like the end sample, are you doing cross validation or what kind of methods are you using to kind of get out of that, like just looking at the in sample data?

It's a really good question. So we do, we do out of sample, we do hold out tests is what we call them at Recast. And I'll talk a little bit about what that is in a second. But first I'll talk about cross validation generally and where it works for MMM and where it doesn't. So the idea behind cross validation used a lot in machine learning, very important method in machine learning is you basically fit the model to different subsets of the data and then you test the model's predictions on the subsets of the data that the model hasn't seen, the holdout subsets, and you do that on a bunch of different subsets and then you can use that to either combine results from different parameter sets from the model or use it to choose a model or evaluate how well the model predicts on data that it hasn't seen before for very important, very powerful method. The thing about MMM is that standard cross validation where you just randomly choose different dates to include in the model or not, doesn't make sense because MMM is a time series problem.

Yeah, time series always, always throws the wrench in there with, with cv.

That's exactly right. Time series causes a lot of problems for cross validation. And so if you just use a standard cross validation algorithm, you're not going to get very good results. And this is in large part because, excuse me, time series are correlated. And so if you tell me what you know last week's results are and what next week's results are, but you drop out this week's. I actually have a lot of information about this week because what the marketing spend this week is correlated to last week and next week. The revenue this week is correlated to last week and next week. And so just like randomly dropping out results doesn't really help you. What you actually need to do is you need to do hold out, hold out testing. You need to run the model up to a month ago or three months ago or six months ago, and then predict the next six months, three months, one month on data that the model has not seen. Right. That the model can't know anything about. And then that is a really good measure of actual forecast accuracy. And that's the thing that you care about. But even beyond just forecast accuracy, which is important, what actually matters is forecast accuracy in the face of budget changes. Right. If marketing budgets never change, forecasting is really easy because you just sort of everything continues forward. The problem is, okay, last month we spent $500,000 on Facebook. Next month we're spending $250,000 on Facebook. Can the model figure out what's going to happen when we actually make these big changes to our marketing budget? And so that's the thing that you actually need to look for is can we predict out of sample right into the future on data the model hasn't seen before? And can we do that when marketing budgets are changing? And so at Recast, we built a bunch of machinery to evaluate a couple of different things. So first of all, constantly running these, these backwards holdout tests, we always run at least a 30 day holdout every time we refresh the model. But then also we're always evaluating how hard was that prediction. Right. How much did the budgets change over this period? And therefore, you know, if it's a really easy prediction because budgets haven't changed very much, we don't give ourselves much credit. But if it's a hard prediction because Facebook went way up and TV went way, way down or whatever, then we give ourselves a lot more credit because that's a really hard prediction to make. And so that we think is the right way to evaluate a marketing mix model that isn't susceptible to the problems of overfitting and really being able to game the metrics. That's a very difficult metric to game.

Yeah, yeah. And there's, we're talking about validating the model after it's been Built after we've already selected our data, carefully manicured it, but there's so much that goes into it before that. It's what data are we feeding into it, what transformations are we making? Are there errors in the data? There's so much that even before you get to the actual model building and testing, that needs to happen to make sure that the old adage garbage in and garbage out. We have to make sure that we're. That the data set is actually clean and Good.

That's right, 100%. And this is often the thing that stops a lot of people from getting started with marketing based modeling is they just don't have the right data. With modern digital platforms, it's not that hard to get access to the data that you need. How much did you spend? How many impressions did you get on Facebook or Snapchat or whatever? But there are other channels that are a lot more difficult things like influencers. We see this a lot with people who work with influencers. Whoever is running the influencer program just like never updated the spreadsheet and so we don't know what happened back in time. Right. It's a huge pain to go figure out and find that data.

Or you know, if you have to get a bunch of PDFs of invoices from out of home marketing and have some manually enter them, those things are a huge pain.

And this is my advice. My advice to every marketer or every CMO is like start collecting the data early. Right? You will never regret knowing how much you spent on out of home and these channels over time historically, like you will want, there will eventually be questions that you want to answer that require that data, whether that's mmm or something else. And so you should start collecting it now. Especially for these like really annoying channels where someone is going to have to do something manually. Giving someone a file of like 108 PDFs from the last couple of years of invoices, that is terrible. But if it's just someone's job to like plug the number into a spreadsheet every week when they pay it, that's not so bad. So like start doing it now is my advice to everyone. So that way you have the data in the future when you actually need.

It or you know what, you know, this is something probably in the next day or two. I'm sure we could use ChatGPT or GPT4 or whatever to just like do that for us. So eventually it'll be there eventually it'll be there eventually.

But you also need to Track the right data. And this is another problem. Like podcasts, we've seen some agencies, they don't track when the actual podcast goes live. They just bill every month. And you, if you want to do mmm, at least, like, you need to know when is the drop date. Right. And how much did you pay for that particular instance? Just knowing how much you paid for all podcast sponsorships over a month isn't really enough. And so, like you, we, we need to push these agencies and push our partners to be able to give us the granular level data that we need in order to have accurate reporting, first of all, and then be able to answer these more sophisticated questions like we do with mma.

Yeah. Or like direct mail. We've had a client that they were fortunate enough that not only were they able to just say, like, they, they weren't able to just say, okay, we spent this much on this month and direct mail, but they were able to know, like, here's when we sent it, but also here's when it actually got to the mailbox.

Yeah.

Which is fantastic. Not everyone is in that situation. But yeah, it's. It's understanding the nuances of the channel, which, you know, especially as a lot of digital markers marketers, we may not necessarily have that kind of level of understanding of how different channels, especially offline channels, are, you know, different in significant ways than what we were used to.

And this is another hard thing about mm, because the idea. So the idea behind mmm, once you get to the advanced levels is you're really trying to encode the way that the real world works into some statistical model. And an understanding of how podcast is different from TV is actually really important. Right. The way that people consume podcasts, there's a drop date, but then people don't. Then people download it, but maybe they don't listen to it. And then there's this long tail. Like, that is a really important thing to understand in order to get a correct model. If you assume that the consumption is the same for a podcast as it is for tv, your model's not going to match the real world. And so a really deep knowledge of all of the channels is actually really important to building a good. Mmm. Because the goal is to have a model that captures all of the relevant facets of reality in the statistical model. And so you need to know, okay, how do these different channels work? How can we capture that statistically? And then to the extent that we can't, like, where are the assumptions that we're making and how do they not line up with the world. And then what does that about how the model might be wrong in the future. That stuff is really hard. And it requires both like statistical expertise at a very deep PhD research institute level and then also marketing expertise. You know, you have to be on the ground talking to marketers, understanding deeply how all of the different channels work in order to go to bring those pieces together.

Exactly, yeah. So one of the things that we've talked about, like the data side of it going into the model, we've talked about, you know, testing, you know, metrics that we look at for testing whether the model is accurate or not. And one thing that I've been seeing get a lot of play recently is the idea of, you know, triangulating different sources of truth. Right. Looking at not just marketing mix modeling or multi touch attribution, but running experiments to sort of validate one or the other and maybe talk a little bit about that, about how marketers can use randomized controlled trials or experimentation to kind of get that ground truth to either validate what the model is saying or maybe feedback into the model.

Yeah, this is a really great question. So at Recast, we are totally on board with triangulation and very much think that the best marketing teams use all of these tools in order to bring the whole story together. All of the tools have structured strengths and weaknesses, and so you sort of need all three in order to get the clear picture from all sides of what's actually going on. Experimentation in particular is a really interesting topic. As a causal inference researcher aficionado, fanboy, I spent a lot of time thinking about experimentation. Experimentation is great in marketing. And there's really two types of experiments that people run. So one is randomized control trial where you run the experiment at an individual level. You take a group of people, you split them in half. One half gets ads for your business, the other half gets ads for the Red Cross, let's say. And then you compare the purchase rates between those two groups. Works very similar to like a medical randomized control trial to test out a different drug. The other type of experiment is a geo lift experiment for non addressable media where we can't identify individuals and therefore be able to treat them individually. We need to do it at a higher level. And so, you know, you run TV ads in Atlanta but not in Houston. And then you compare the relative purchase rates at the geographic level between those two, between those two different geographies. So those are the two different type of types of experiments. The randomized controlled trial at the individual level is best when you can do it. But even then, right. These sorts of experiments have limitations. What are the. So before we get into the limitations, pros is that this gives us a really good sense of causal inference. If the experiment is well run, we actually do get a really good estimate of incrementality, the thing that we actually care about. However, a couple of issues. So one is that it can be difficult to power these experiments really well, depending on how fast moving your product is, what your conversion rates are, et cetera. Right. So they can just be difficult and they can often you can get a result where it's like, there is a statistically significant lift, but the question is always, well, how much lift is there? Right? Is it a 0.5x ROI with a statistically significant lift or a 2x ROI or a 5x ROI? And a lot of times you get results back and it's like, well, it's between 1x and 5x, which is like, great, it's a positive, but we don't really know what, like what the ROI is and therefore how to allocate our dollar. So there's precision questions that can be a problem. And there's also the issue that these are always snapshots in time. Right. You run a test between April 1 and April 15, you get the results back, and then every day you are outside of the window when the test was run, you have to trust the results a little bit less because that was run at one very particular point in time with the creative you were running at that point, with the offer you were running with the website that you had at that time, with the way the platform worked at at that time. And all of those things are constantly changing. And so when it's a month or two months or six months later, you always have to say like, well, do we still trust that? And we don't really know. And that's the big problem with experimentation is you can't. It's difficult to bring. It's difficult. You can't run a test in every channel every day, all of the time. And so you need some way to bring the story together. And mmm is a really good framework for that.

Yeah, yeah. We had a client that we were proposing to do a branded search test to see, you know, they wanted to decrease branded search and we wanted to test out to say, okay, how is this something that is incremental or not? And they had a little bit of pushback because they said, well, you know, our counterparts in this other country ran this branded search test three years ago and here's what they saw. So we don't want to test that. I'm like, first of all, it was three years ago. Second of all, it was in a different country. Like, yeah, I mean, you have to. You have to have this sort of organizational mindset of test and learn and constantly be testing and sometimes, you know, test the same thing that you did a year ago. Because it's probably changed or more than.

I mean, or less than a year ago. Here's a long. A year ago. Was Covid still happening? Like, the world was totally different a year ago.

Yeah.

And so, like, the things.

Interest rates were 0% and yeah, it's like the whole.

There was no inflation. Inflation, or we had just started inflation. Like, it's a whole different world. And so being like, oh, we ran this test a year ago, to me, is crazy. The world changes so much faster than that. And the platforms themselves change. Facebook's algorithm changes. Instagram introduces reels. Like, I don't know, there's all kinds of stuff that happens and you need different ways. You need to be able to stay up to date on those measurements. And more than a test of more than a year ago, I would basically throw out. It gives some information, but it's not a ton of information about what's actually going to be happening in that channel for your business today.

Exactly. Yeah. So let's assume that someone just went out and they sell, oh, hey, Robin lets me do marketing mix modeling. And they, they go and they put their data into there and hey, I got this. I got this model that says I should increase Facebook by, you know, 28%. So they dumped their budget into Facebook and they decreased their budget and podcast episode advertising, by the way, if anyone would like to advertise on the podcast. So what are the costs of not validating your results? What if you're wrong? What if the model is just completely unbased and the results that it's showing you?

Yeah, I mean, it's easy to imagine. Right. The costs of being wrong with any sort of attribution or incrementality estimate are very high. Right. If you put a bunch of money into a channel that is not actually performing very well, that money is wasted effectively. And so there's the dollars themselves and there's the opportunity cost. If you could have been allocating those dollars to some other channel. I mean, when with the customers that we're working with, those optimizations add up to millions of dollars a year easily. Right. And so the thing that you're paying for with an MMM is not to just get results. Right. It's very, it's trivially easy to get results. The problem is that they have to be right. And there's no easy way to validate whether or not those results are right. And so whether it's Robin or some Google sheet, right, if you get those results back and you go out and you dump 100,000 or 500,000 or a million dollars into some channel on the back of those results and you don't get a solid ROI for those dollars, I mean, those dollars are wasted. That's a huge cost, way more than whatever you were paying, like for the MM itself or for the analyst to go and run that. And so when I'm talking with people about doing an mmm, I'm like, to some extent it's better to not do an MMM at all than to do an MMM poorly, because the MMM poorly will actively misdirect you and you don't have any way of knowing until the dollars are already wasted. So, like, if you're not going to do it right, and if you don't know what you're doing, best not to do it, like stick with MTA or whatever, because at least that won't actively mislead you. If you go down the MMM path, you need to know causal inference or you need to be working with someone who does, who can think about these things and make sure that they're not going to be leading you down the wrong path. They're not going to be taking that multimillion dollar budget, multi tens of million dollars, multi hundred million dollar budget, and then allocating those dollars based on effectively nothing, based on ghosts in the statistical machine. And that's where the danger of MMM actually comes into play.

Yeah. And making sure that you're not using MMM as a crutch, as a. Well, the model said this, so this is what I have to do. But not ceding your decision making to a tool or a model or something like that, but just saying, okay, model said this, okay, that's, that's some information I can use to make a decision. Maybe it says to decrease spend and TikTok by 10%. Okay, maybe I'm going to decrease by 5%. Right. And so a lot of times I have to kind of guide clients through this. Like it's just a model. It is not ground truth. We can use it to help us make decisions. Maybe we test some of those decisions. But not just saying that, okay, this is what the model said. Let's go ahead and do it.

I think the test and learn approach is, I think, the right way to approach it. And this is, again, a thing that we really emphasize at Recast is we say, like, don't take our word for it. Right. We don't want you to just trust us. We want you to actually go out and validate the model's results and do that actively in a way that is testable. If the model says this channel is underperforming, let's pull back on that channel for a month. Right. If you're right, revenue will drop a bunch and we'll turn it back on and it'll be okay. Like, we can always do that. But if we're right, your revenue is going to stay flat and you're going to save all of that money. We just got through a big test with one of our clients doing this with one of their channels. Results came back perfectly exactly as we forecasted. They turned off the channel, revenue flat, thumbs up. Now they're going to save. You know, I think it's like high six figures a month going forward just from pulling back on that channel. And this idea of, like, let's go test it. How can we, like, having a debate about, you know, oh, I don't know if I trust this assumption or not. That's not actually that productive. A productive way to use the MM is let's go test it. Let's go run a randomized control trial via Facebook's API. Or let's go run the GEO lift test. Or let's just double spin this channel. Next month. Recast says we're going to get a bunch more revenue. Let's see what happens. If it comes true, great. We have a winner. If not, that's okay. Like, we maybe spend a little bit more money than we wanted to, but now we know we can adjust the model and we can move forward. And so that's very much what we encourage with our customers, is that testing and, like, really going out, thinking in bets, not getting locked into. Oh, man. Just because we made this budget six months ago, we really have to stick to it. We really want people to, like, go out and test, experiment, and then use that to drive the business forward.

Yeah. And it's so. It's so scary for marketers because they, you know, from their mindset, they're saying, if I. If I make this bet and it doesn't go right, if I put, if I double my budget in this channel and nothing happens, oh, that could be my job on the line. But that's the wrong kind of thinking. And hopefully, if you're at an organization where that's a concern, then maybe you're at the wrong organization. But, like, you're still getting even if you double your budget and you get, you know, very little return on that, the learnings you get from that. Right. You're paying to reduce uncertainty. Right. There is dollars and cents value to that reduction of uncertainty because now you're going to make smarter decisions going forward that will make you or save you millions of dollars.

Yeah, and that's right. This, like, thinking in bets idea is, I think, really important to be successful as a marketer when the world is changing as much as it is right now. And so this idea of we have to go out and learn, we have to if we want the business to grow. Right. If we want to be able to expand into new channels and hit that goal goal for the end of the year, like, we can't just keep doing what we're doing. We're going to have to try new things. And you're right that, like, for a lot of people, feel like their job is on the line. And that's unfortunate. Like, we need to help educate organizations as a whole that we have to go take bets as marketing teams, as marketers. And that means we can't, like, fire people when, like, one bet doesn't work out. We have to build up this culture of being able to run experiments. And I mean, again, if you, if we run 100 bets and none of them pans out, like, that's a real problem. But, like, just because one doesn't work doesn't, you know, that's actually like a good thing. We learned something. Let's go try something else. And if you can do that consistently, you're going to be moving up into the right. And so the goal is to really make smart bets. Where can we get signal? Mmm is one way, experiments are others. The MTA is another. To go make those smart bets and drive the business forward.

Absolutely. All right, Michael. I like to end the podcast with an action item for my listeners. One thing that the marketers and analysts listening can do immediately after they leave a rating and review for this, for the show. What's one thing after they do that? What's one thing that they should do right now, right after they finish listening and they can take action on.

So I think the one thing that marketers, the one thing that I would love every marketer to be doing is just really thinking hard about incrementality. What does incrementality mean? How do we think about it? In our business, how do we measure it and then how do we use those results to drive the business forward? That I think is the most important concept in marketing right now. And it's gained a lot of traction over the last couple of years. And I think probably the, the marketing nerds who are listening to this contact, listening to this podcast are probably familiar with it. But go out and evangelize, go teach your friends. And to the extent that you're not really using this idea of incrementality within your business, I think that's the thing that everyone should really be going out, learning more about, learning about how to measure it and then really using it to drive decisions going forward.

Absolutely. And it's such a mind shift for again, especially for digital marketers who've for the past decade plus have been into this sort of multi touch attribution. To go from that where we're saying here's, you know, something that has a sale that happened, let's give credit for that sale back to these channels. Same for this backwards looking, giving credit to now saying okay, well actually what was. Would this purchase have happened even in the absence of marketing? What's the incrementality? Right. And that's such a change in mindset. I feel it's. It took me a long time and I still struggle with, I still kind of slip back into the, you know, the attribution mindset. Sometimes I have to kind of fight out of that. But yeah, I definitely agree with you. To encourage everyone to really start to focus on incrementality and how we can understand incrementality better. And one way that they can do that is by going to getrecast.com and subscribing to the weekly newsletter from Recast, which is a gem, comes out every Wednesday. I look forward to it. Tomorrow it's going to pop into my inbox and I always look forward to. Because you guys have great content in that newsletter. I usually steal some of the stuff that you put in your newsletter and put it into the the Mix it up newsletter from MMM Hub. So it's a wonderful sharing of ideas here. But you guys always have some good stuff in there and on your blog too. But since I've already pitched your newsletter, what else would you like to pitch or where can people find you?

People? People can find me on Twitter at Mike Kaminsky. Follow me on LinkedIn. I talk a lot about these topics on LinkedIn. That's probably the best place to follow me. You can find me at Michael Kaminsky if you search Michael Kaminsky recast. I'm sure that I'll pop up. Yeah. And any listener who wants to nerd out on marketing, measurement, incrementality, marketing mix, modeling, shoot me a note, would love to chat about it. I'm obviously super passionate, always looking to learn more and always looking to talk to interesting people. So please reach out. Don't be a stranger. And I'll be looking forward to meeting y'. All.

All right. Thanks, Michael. Thanks for being on the show, taking your time to help educate us and help us make better marketing decisions.

Absolutely. Thank you, J.