Media Mix Modeling and Finding Incremental Revenue | Michael Kaminsky (Recast, Harry's)

Foreign.

Welcome to the Efficient Spend podcast where we help marketers turn media spend into revenue. My guest today is Michael Kaminski. Mike, thanks for being here.

Thanks for having me, Paul. I'm excited to be here.

Yeah, I'm definitely excited to chat with you about your experience at Recast and also just media mix optimization more broadly. I think it would be interesting just to give the audience some context into where you got started though, because a lot of folks in the marketing space don't necessarily have this like foundation in econometrics. So would love if you could kind of start with like the early days of your career.

Yeah, for sure. So my background is in economics and econometrics. I got my start actually as an undergrad. I took a bunch of economics and econometrics classes and then had had the, had the privilege of getting to do like RA research assistant work with some professors there. So I got to work on like doing real research, publishing papers. From there I moved into consulting against doing like academic style research. So like having a problem having some data and then having to come up with some way of getting interesting learnings out of that data and then writing them up for, for publishing. So my background has always been on this idea of like statistical inference and in particular causal inference. And when I got, when I got started actually I think we like the world wasn't that good at causal inference at that point. Like there were still, we were still using a lot of outdated methods. But over the last 15 or 20 years like our ability to understand how can we use data and statistics to learn things about the world has gotten a lot better. And I've been, you know, feel very privileged to have gotten to go along for that ride. And so I left like the academic consulting style world and moved into startups where I got to work with a lot of growth marketers. And so I came into marketing through science of. You know, when I first started working at Harry's, working with the marketing team there, I didn't know anything about performance marketing at all, but I knew a lot about science and statistical inference. And so I got to have a really good time working with the team there to think about, okay, we have this data, we have these really hard questions that we want to go answer. What's the best way to go answer them? And so that's, that was sort of my journey into marketing science was very much from an academic background and then paired with like this desire to actually drive a business forward at Harry's of hey, we, you know, we have these goals we have these problems. How can we actually go and make a difference in the business? And so I'm, I'm very passionate about this idea of taking things from, from the academy, from sort of deep science and then applying them to real world problems.

Sure. Essentially analyzing the data and then actioning on it and coming up with a strategy and then a level of assumptions making experiments and then seeing that play, play out in a real business environment. So at Harry's you were the director of analytics from July 2015 to roughly July 2018 according LinkedIn. Roughly.

Sounds about right. It's been a while now, but sounds about right.

That's a super fascinating time at Harry's. Right. So maybe just give the some context into kind of like that scale that happened in that three year period if you want to bake that in. I don't know if you can give kind of like acquisition numbers or revenue numbers or whatever, but to me I, I don't have that much context, but I believe that Harry scaled a lot in that three year window.

Yeah. So I don't actually even remember any of the actual numbers, so I won't be able to share them with you. But there was huge amounts of growth over that time period. Step change growth. 2015 to 2018 was like an exciting time for the world. There were a lot of developments happening, especially on the performance marketing, e commerce acquisition. There were a lot of businesses that were growing around that time and so it was a very fun time to be there for. For Harry's, the growth came actually like not just from the e commerce and digital side of the business, but from opening up channel partners with major retailers. So during that time, Harry's launched at Target in store and sort of did like an initial limited rollout and then all of every Target nationwide in the United States. Same thing with Walmart, same thing with grocery like Kroger and other grocers. Same thing for drugstores like Walgreens and you know, basically anywhere that you can, you could imagine buying a razor blade. Harry's expanded distribution channels there. And so that was a really exciting time to be there because like a lot of the fundamentals of the business really changed over that time period. The way that you think about marketing, if you're an E commerce only business, is very different than the way that you think about marketing for a business that has distribution through all of the different retailers. And so I was, I feel very privileged to have gotten to, to go on that journey with a business that was growing so rapidly but also fundamentally transforming itself as part of that growth.

Sure. Much of my experience in scale is kind of staying within one funnel, meaning a website or an app, and then scaling into different channels. But it Harry's, it's scaling into different ways that you can acquire a customer or different places that you can acquire a customer. And, and as a director of analytics there, could you kind of dive into your roles and responsibilities and how you were looking at that and measuring that performance?

Yeah. So my team sort of did everything and we had a very lean team. We did everything from managing the data warehouse and all of our BI tools, so basically reporting for the entire company. But we also worked on developing, you know, machine learning algorithms to do churn prediction and help reduce churn from our subscription product to doing statistical inference, you know, helping the team run a B test. We built some internal tools that helped product managers run and interpret a B tests. We. And then that all, I think is probably dwarfed by the work that we did with the marketing team on helping the marketing team measure the effectiveness of their marketing spend. And so during my first year there, spent a lot of time thinking about digital attribution. Right. So digital tracking plus post checkout survey. And how do we map that on to our decision making process, you know, ltv forecasting, combining that together. And then once we started launching in retail, thinking about, well, this is a whole new set of problems, how are we actually going to, to estimate the effect of our marketing spend on retail where there's no tracking at all? Right. Someone, you know, sees an ad on Facebook or listens to a podcast and then grabs Harry's off of the shelf. Is there a connection there? How can we draw a line between that? How can we make smart decisions about our marketing dollar allocation in that world where there's no pixel, there's no tracking that's happening at all?

Sure. And how did you kind of approach that, that problem? And what were some takeaways? I mean, I guess, you know, I don't know how much context you have into kind of the, the, the media mix and the components of that. But how did that shift over time as you were going into those different retailers? Were you doing more geotesting, relaunching into different channels? What are some takeaways that marketers can kind of get from your experience there?

Yeah, so definitely, definitely doing. I mean, first of all, this was just like a big problem that, that the whole company was like, oh man, this feels like a whole new thing that we don't really know how to do. I remember I had a Conversation with my boss at the time, who is the Jeff Raider, the CEO of Harry's. And, and he was like, look, you know, we all sort of know that what we're currently doing of just measuring marketing effectiveness on the E Commerce side isn't right. Right. But we have no idea what we actually should be doing in order to optimize our marketing mix for the holistic business that includes E Commerce as well as these other distribution channels. And I was like, I don't really know either. So let's go figure out what we want to do. And so we started doing some experiments like lift test, like match market, geo geo holdout, lift test, style experiments in order to get a read on. Okay, you know, when we increase spend in, in these marketing channels in these certain GEOs, can we get a read on how that impacts our offline sales? And so that was sort of dipping our toes in the water of trying to understand this phenomenon. And what we learned there was that like, yes, we can get a read on it. Yes, these marketing dollars do drive in store sales, but we're going to need, in general, a more sophisticated method for thinking about how to bring all of those pieces together. Geographic holdup tests. They're generally what a statistician would call underpowered, which is to say that they're not very sensitive. It's very difficult to tell, like, is the ROI 1x or 1.5x or 1.75x.

Right.

It's very difficult to differentiate between those because the underlying data is so noisy. And so you can see that, hey, there's some lift, but there's wide confidence bounds around it. And so we're not really sure what the true number is. Plus other problems of not being able to measure them consistently. Right. And for a business like Harry's, you know, the holiday season is very different than the summer season, is very different than the spring season. And so, you know, if we ran a test in the spring, can we apply those results to the holiday? We had all of those problems. And so we knew that we wanted to think about more sophisticated ways of being able to bring that information together and then make smart bets about where to allocate our marketing dollars in order to drive the business forward. And so that was where we started exploring ideas of, you know, doing lift tests, combining them with marketing mix modeling, and then trying to have some more holistic way of driving the business forward across offline marketing channels as well as offline sales channels.

Did that analysis kind of uncover any gaps in the media mix that, for example, you were underspending in different offline channels or overspending in different areas.

Oh yeah, absolutely. And I mean, this is a thing that everybody sort of knew was the case, right. That if we optimize our marketing mix to the channels that are most optimized for e commerce sales. So in general we're talking about like bottom of funnel digital channels that are very easy to track. So your Facebook, Google. Yeah, Facebook, Google. Right. Like those two channels were great for e commerce businesses and they're optimized for that, but they're probably not actually optimized for driving sales at Target and Walmart. And so it became very clear very quickly that actually like we were sort of upside down in terms of how we were optimizing our marketing spend because we were so optimized towards the things that were best at driving sales in the e commerce site, but not necessarily things that were best at driving sales across the whole business. And, and again, I mean, I don't remember the actual details from Harry's, but across all of the different businesses that I've worked with over the last couple of years, you know, the things that are obvious start to jump out. Right. More offline channels like radio are often not as good at driving e commerce sales, but a lot better at driving sales in Target and Walmart because people listen to the radio on the way to the store and their broader reach channels where you have broader distribution, it makes more sense. And so there are a lot more. Once companies have more distribution outside of just E commerce and they have more ways to sell to customers, opening up some of those broader reach channels can actually be really, really impactful.

Sure. Even things like billboard ads, right?

Billboard ads, other out of home, all of the stuff that again is, I think, really hard to make work if you're a pure e commerce business, not impossible. Like I've seen E commerce and digital businesses make out of homework very well. But those sorts of things become, I think, easier to make work. And you know, you can tell a story in your head about why again, you know, billboards people are in the car, they're going to Walmart, that's a great time to hit them. Why that might actually work. And so again, these companies as they make the transition, they want to start playing with these other marketing channels. But the measurement, measurement is really hard. Like it's really hard to make a connection between the billboard that someone sees, but there's no impression registered in a pixel. And then when they go to Walmart and grab something off the shelf and then pay in cash. Right. There's just like no way to really make that connection outside of some experiment or some statistical methodology for looking at, on average, what's the impact of this type of marketing investment?

Let me ask in the Harry's example, I think, you know, something that I've kind of found when, when chatting with founders is that there is this sense of almost like an addiction to digital attribution in a way. Right. Prove to me that this marketing spin was effective at your experience at Harry's, I guess, in the past. Like how, how do you respond to that? When someone comes to you and says, hey, you're telling me that I should stop spending in this area that has a very good looking CPA and go to this area, like offline that I can't measure as easily?

Yeah. So it's funny. I mean, first of all, at Harry's I was super lucky to work with a team and a leadership team in particular that was incredibly, incredibly smart and very, very thoughtful about incrementality and the things that we actually care about. So with the leadership team there, they always got it immediately. Like they were never, they always realized, hey, look, click based tracking is going to be biased. And so we're very willing to look at other methodologies that might have other and different biases, but it's not like we're addicted to that. That being said, the phenomenon that you're talking about, I have also seen in lots of founders and marketers that have come up in the digital tracking world where they say, oh, I don't know, you know, I can actually see the click on the Facebook ad and therefore I know that it's working. But the problem is that that click doesn't mean that it's working. Right. Just because there's a click or just because there's an impression doesn't mean that, that, that that ad actually caused the person to make a conversion. The, the branded search example is the one that people always come back to. That I think is very obvious for people. And it resonates, which is if someone is searching for your brand, right, they're searching for Harry's razor on Google. If they click that first paid search ad, does that, did that paid search ad cause them to buy the razor? Well, probably not. They already knew about the brand, clearly. They clearly wanted to go to Harry's website based on their activity. And so if you say like, oh man, we had to Pay, you know, $2 for that click. We've got a great ROI though, because they always convert. It's like well, not really, because most of that spend probably isn't incremental. And so that logic makes a lot of sense in branded search, but it's also true in Facebook. Not all of your Facebook clicks are driving incremental conversions. Facebook is really, really, really, really good at showing ads to people who are already very likely to convert. And that, that part of the sentence, already very likely to convert means that not every one of those clicks is truly incremental because they were going to convert anyway, or at least some portion of them were. And so once you sort of get that lesson into your mind that a click is not does not mean that it is 100% incremental, that it, that that advertising property alone actually drove the conversion. Well, then you start to question everything. And then your mind can open up to this idea that, well, maybe clicks aren't, aren't everything. And we actually need to think about true incrementality through experiments, through other methodology that can lead us to actually make good decisions for the whole business, which requires thinking about this incrementality question.

Sure. You have an excellent article on, on Reforge called the Attribution Stack that goes into this in a little bit more detail. Basically all of the different methodologies and ways that you can measure marketing spend and where certain things work more effectively. I do want to ask though, you know something that a lot of the digital channels are very good at? Are there kind of like bidding and optimization machines that are heavily based on different data, that data is slowly going away? What are the implications of that from your perspective?

Really good question. So, I mean, I've read a bunch of articles about this. I don't think anyone really knows what's going to happen here. I think working from, from first principles, what you expect to see in, you know, Facebook and Google and losing ability to target as well as they have before is that you're gonna have to show more impressions to get each different conversion. Right? Because you're not able to target as well. And so if you want to sell something, you used to only have to show 10 impressions because you were showing it to a very narrow group of people and now you need to show 50. So if everyone needs to show more impressions, the market is going to cause the cost per impression to go up. In general, your marketing is going to get less efficient. Right. So across all of the different platforms, we expect to see marketing getting marketing dollars getting less efficient as the ability to target and track gets worse. Beyond that with within any individual company, we expect to see, and we have Already started to see a lot of this, which is that their digital tracking and attribution technology is going to get worse and worse and worse. And so, you know, one metric that I ask a lot of digital marketers to look at is, hey, what percent of your conversions actually have tracking associated with them for that conversion? So whether that's last touch or first touch or whatever, what percent of those conversions have that tracking associated with them? And then I ask, and how has that changed over time? And for a lot of people, it used to be like, oh yeah, you know, a couple of years ago, 70% of conversions happen had that tracking associated with them and only 30% didn't. And now that number is like 50%, 40%, 30%, depending on, you know, what sort of audience you're, you're advertising to. And so, and then I asked them to think about the implications of that. And I think that those are the things that are really important, which is that one, we should expect that our digital are sort of star channels like Facebook and Google. They're going to get worse over time as their ability to target individuals gets worse. That means that they're going to be more on par with other broad reach channels like TV or linear TV or streaming TV or Netflix, which is maybe streaming TV, I don't know how we count that. Or YouTube. Right. And some of these other channels. And internally at our business, we're gonna have to think a lot harder about how we do our reporting because one, we're going to diversify our channel mix and two, we're just going to be able to attribute less of our conversions overall to those different marketing channels. And so we're going to need other ways of thinking about how to allocate our marketing dollars across. Across the mix.

Sure. And the ultimate kind of solution that you come to is media mix modeling, essentially.

Well, I mean, look, I think they're all. I think media mix modeling obviously has a role to play for a lot of different businesses. So that's what we do at Recast. Hi everyone. Recast does media mix modeling. And so I think media mix modeling is important, but I don't think it's the end all, be all, and I don't think that it's right for every business. And this is a thing that we, that I, I give advice to a lot of different marketers at a lot of different stages of different businesses. And what I tell them is that there's no magic bullet, right, Ever. There's no tool that you can buy that will relieve you of the requirement of thinking hard about how your marketing is working for your business. So unfortunately, every marketer needs to spend a lot of time thinking about what is each measurement tool doing, what is it contributing to our understanding of incrementality in our business and then how are we going to make decisions going forward. There's a lot of small e commerce businesses that are, as long as they're sort of pure e commerce like owned e commerce, Shopify store, they don't really need anything outside of basic digital tracking, like what comes out of Google Analytics or what you can get for free out of Shopify. You can build a great little business just based on that without doing too much otherwise. And I think that's awesome. And I would never try to sell recast to a business like that. But as businesses get more mature, then they need to sort of start layering on other types of technology. And so it might be like, okay, we're going to, you know, do last touch attribution and we're going to do a post checkout survey. Okay, great. That gives us another data point. Post checkout survey doesn't measure incrementality but can give you some ideas around like other things to experiment. Then from there you can start layering on doing real experiments. Right? You can use the Facebook's internal tool for doing a lift test. You can do match market geography holdouts. These aren't that difficult to run in the major digital marketing platforms today where you just tell like, you know, I don't know, Pinterest, like only show ads in, you know, the western half of the United States and not in the eastern half of the United States. And then see and then come do a pre and post analysis of what happens to your sales when you turn off ads in half of the country. These are like basic analyses that you can do in Excel. You do not need a statistics degree in order to analyze and experiment this way. And so I would encourage businesses to start to do that, to really build that testing muscle of, hey, we can run a test, we've learned how to interpret it, we know how to think about it and action off of it on a business. And then once you're doing that consistently and you're spending, you know, millions of dollars a year on marketing, that's when you want to start thinking about other more sophisticated tools like a marketing mix model and mmm to bring on to help pull all of those different pieces together. But again, I think that there's an evolution that companies go through and marketers need to be really thoughtful about what is the right tool for the business at the stage that we're at today and the decisions that we need, that we need to be making.

For sure, the more noisy things become, the more challenging it becomes. And I think for a company that is just getting started, basically adding any amount of marketing spend is all going to be net new. It's all going to be easier to kind of measure. But when you have a business that has been around for a while, that is running multiple channels, that has multiple ways of acquiring customers, multiple funnels to acquire customers, that's when things definitely become more complex. Let's get into Recast. You're the founder of Recast. You have a pretty strong client roster. Masterclass away rocket money. And you're definitely kind of like solving this problem of less. Less tracking to. To work with using a kind of older technique from the. From the Mad Men days. But you also kind of differentiate Recast a little bit. Talking about kind of Bayesian. Sorry if I'm pronouncing that correctly.

You nailed it.

Perfect marketing mix modeling. So, yeah, what is. What is unique about Recast? You're playing in a pretty competitive space right now, and I'm sure that's a loaded question, but I'll give it to you and let you try to attack that.

Yeah, no, this is. This is great.

So.

So first I'll clarify. Co founder of Recast, my partner, Tom Vladeck. He is the mad genius behind our underlying statistical model and a great co founder. What makes us different? So there's. You're right that media mix modeling has a long history, and it comes from a world where there was no E commerce. And basically, if you worked at a CPG company, let's say you were the CMO of pepsi, and it's 1985, and you need to decide how much TV versus how much radio versus how much like print ads you're gonna buy in 1986. How do you go about deciding that? And what you would do is you would hire a team of statisticians to look at your historical data. Try to look at, okay, when we spend more money on TV versus print, what happens to our sales? And then you go into your sort of upfronts, right? Which is the way for the young people on the call is the way media used to be bought is twice a year, you would go to an auction with the other big brands and buy a bunch of TV ads and buy a bunch of radio ads. And that was how it worked. And so that process made a lot of sense. You hire the Statisticians things don't change that much over time. You look at your historical data and you buy media twice a year so that like, a whole industry developed to provide that service. Starting again in, like, the 60s, effectively. And then over the next 40 or 50 years, people continued to use the same methodology with slight refinements. The problem is that in today's world, that style of consulting, once every six months we're going to do an analysis, doesn't really make sense for the way that media is bought today. Most company, I mean, no company that recast works with, buys media and upfronts. Everyone is dynamically bidding on media, even for things like tv. There, we're dynamically bidding. Things change really rapidly over time. There's a pandemic and cost per impression goes down or up. Demand for our services goes down or up. Things are just changing so much. So this idea of, like, we're going to, you know, do this analysis, use that to make decisions for the next 12 months, doesn't make any sense. So at Recast, basically what we wanted to do is we wanted to build an MMM that could be updated every week, right, to give marketers continuous insight into what's actually going on in their business. And we wanted to do it in a way that was statistically sound. After doing a bunch of research on the existing players, what we found was a lot of snake oil, Honestly, a lot of consulting firms that had built their business around telling CMOs what they wanted to hear instead of what they needed to hear. And so we wanted to build Recast very differently, ideally for performance marketers who are not trying to just make a case to the CEO and do some back padding, but really are, like, scored against how well do they drive the business forward. And so we wanted to build an MMM for those people. And so we built Recast. We basically rebuilt the whole idea of MMM from the ground up using modern statistical methods and machine learning. These methods that have been developed a lot over the last 10 or 15 years, they just didn't exist, you know, prior to 2005 at all. So you couldn't do it. And so we were like, look, what would look like if we rebuilt it from the ground up today? What are the features that we would want? How would we want it to work? How would we hold ourselves accountable, right, so that our customers know that we're not actually selling snake oil? And that's really where we focus. The business is taking this idea that's been around for a very long time time, but rebuilding it so that it would work for modern marketers and that it would actually be something that would be testable and verifiable and actionable. And that's really what's at the core of recast.

So I want to ask, I know that there are kind of specific use cases for media mix modeling, and like you said, it doesn't apply to every scenario, it doesn't apply to every company, and doesn't apply to every kind of mix shift. So, for example, you would not leverage media, you not leverage recast for something like conducting a creative AB test or kind of like inter campaign optimizations, things like that. I do wonder though, you know, something I've seen with conducting incrementality in the past is that things change over time. Meaning I think sometimes marketers will conduct analysis and find that a channel is incremental or not. And then they assign in their head this thing is not incremental. And I'm no longer looking at it where that doesn't work. Right. And I wonder, because I don't have as much experience into really like leveraging a MMM tool. How do you account for that? Meaning if I am managing a media mix, I think, you know, Facebook might be the thing that's more incremental this week, but next week it might be Google, or next week it might be tv. So how do those things change over time?

Oh man, this is such a good question and I'm really glad that I get the chance to talk about it. So, first of all, this idea of like incremental or not, I don't think is the right framing at all. Right? It's not like there's like a switch that's either on for incremental or off. It's always a matter of degrees. Like what matters is the incremental return on investment or the incremental cost per acquisition. I think there are very few channels that are truly zeros, right? Where it's like, if we spend a million dollars and we get nothing in return for it, that can happen, right? For sure. But I actually don't think that in most companies that's the situation. It's just that if you spend a million dollars, does that get you 100,000 new customers or 10,000 new customers or 1,000 new customers? And that's the question to really be focused on is what is the true incremental lift of these dollars. And it might be that, you know, your incremental cost per acquisition is really high on this channel and therefore it is not worth investing into it. But that doesn't mean that the channel is not incremental, it's just incremental at a cost per acquisition that is too expensive for your business. This. And so that's the first thing to like realize and get into your mind is it's not like yes incremental or no incremental, it's how incremental is it assuming that everything is to some degree, some amount of incremental. And so from there then the question is, okay, what is our incremental return on investment? Or what is our incremental cost per acquisition for each marketing channel? And then you're exactly right. How does that change over time? And so incrementality, again, is not a thing that's like on or off for a different channel. We ran one test, Facebook is incremental. And now we never have to think about it again. The question is, is what's the incremental cost per acquisition on Facebook and is it the same today as it was two months ago? And that is the question that marketers need to really be asking themselves and really be laser focused on. Because again, these things can change. Facebook can change their algorithm. Right? Facebook can lose the ability to, you know, as cookies go away, Facebook loses its ability to track people across different devices and see what they're doing across different websites. And so you would expect that the incrementality would change. And so what you want to be thinking about is, okay, you know, how incremental is this channel? How is that changing over time? And then what does that imply about what I should do next week or next month? And it's also that last piece that's really important. I see a lot of marketers get really hung up on, oh man, what was going on six months or a year ago doing all of these analyses. But really what matters is what are you going to do next week? The past is in the past. And we can use information from the past to figure out what we can do going forward. But to some extent, spending a ton of time on what happened nine months ago isn't actually a really good use of time if it's not going to drive a decision about what we're going to do tomorrow, sure.

Would I be correct in like assuming that there is that that kind of long term data is input and incorporated into the model. But the real question we're asking about is that more immediate, like, what should we be doing next week? Next month?

That's, that's how we think about it at Recast. Right, Recast. Again, we think about it as a decision Making tool for optimizing the business next week and next month. And again, we report out on what happened nine months ago. But we don't want marketers to get hung up on that. And we actually think that can be a real distraction from the actual business of running a marketing program, which is figuring out how are we going to improve the business tomorrow. And so. Mmm. The idea is you take historical data and you do analysis on that. But really, I think what we are focused on at Recast, what we think every marketer should be focused on, is how do we use that to make a decision about what we're going to do next week, and then how are we going to evaluate if that decision was a good decision or not? And I think that's really where you want to be focused. And as long as you're focused as a business on those things, you're going to be sort of continuously moving up into the right, as opposed to focused on, you know, sort of, oh, man, what Was going on 12 months ago? Was that a good decision or a bad decision? It's like, yes, you can do that analysis, but only to the extent that it serves you for making a better decision next month, for sure.

I kind of think about it as all of these different areas are just kind of like, have this point of diminishing returns, and that's what you're trying to find, that, like, optimal point. And even within a specific company, the size of the media mix can change over time. You might be spending, you know, $2 million last month and you're spending 4 million this month. And so adding incremental budget into a higher volume media mix might not be as effective. So that incremental Facebook spend, adding it into this mix where you've already hit diminishing returns, gets saturated. It's not going to be as effective. So I definitely think about that. I know we have a few minutes left. You know, let me ask at Recast, with some of the clients that. That you work with, if you could talk a little bit maybe about their initial experience and after working with you and leveraging. Mmm. For the first time, there's probably a lot of aha moments. Right. What actions are they taking? Are there any kind of commonalities and kind of like spend culprits that they're like, oh, shit, we need to be cutting this thing, or does it kind of. I'm sure it varies by business a little bit, but yeah, so it varies.

A ton by business. I think the companies that have a lot of success with Recast are the ones that are willing to take a look at their sort of deeply held beliefs and then reevaluate them. You know, we've had multiple companies come in who are like, look, we built this business on catalog. Catalog is our most important channel. It's the most incremental for us. People love the catalog. You know, we're going to, you know, we've got $5 million of budget this year allocated to catalog. It's going to be our most important thing. There's no way that it's not incremental. And then recast results come back and we're like, I don't know, the catalog does not look great. The patterns here are not very good coming out of the model. And so we're like, look, just go run a test, right? Buy some extra addresses, keep them as a holdout. We'll do the same matchback analysis, and we'll be able to see how truly incremental it is. And they're like, oh, fine, but it's not going to work. And then they come back, they run the test, and it's catalog, maybe it's okay, but it's not nearly as incremental as they thought. And they're way over spending in that channel. And so, like, that sort of analysis is exactly the same thing that we see across all kinds of different channels. And I think the people who are really good, who are going to get the most value out of a tool like recast are the ones that are willing to run that test to check their deeply, their deeply held beliefs, run experiments, and then prove it. Like, is it true or is it not? How can we actually go and test this and validate it? And those are the things that we love to see. And again, we've seen this over and over again with lots of channels where, you know, maybe the channel's okay, but at a very small level of spend. And what the business had done was they had, you know, done a small level of spend that looked awesome. They scaled up into the channel, but now they're actually not very efficient at the level of spend that they're at. And so they need to scale back a lot or rethink about how they're managing the channel. And those are the sorts of things that we see all of the time. Catalog, again, for these examples that I talk about actually are good investments for these businesses, but they just need to be at the right level at a specific point. Yeah, and at a specific point. And like, there's also like the frequency, like when you were first Starting the business. And you sent your first catalog. That one had a huge effect. But now you've sent hundreds of cat. Like, you've done this so many times. You're reaching, diminishing returns. You need to rethink about how you're running this channel. And so, again, part of using Recast is actually, I think for the businesses that are really successful, it helps them change the way that they run the business and the way that their executives think about the business in a really positive and healthy way instead of just focusing on, hey, you know, what did the vendor come back and tell us about how successful this is? They start to think about, how can we test it, how can we validate it? What's the true incrementality? And if you use Recast and like that is the result is just that your executives start asking those questions, that's going to be a huge win for your business.

Right? Yeah. I mean, I can speak personally and say that, you know, as a result of Apple's privacy changes, our company has definitely thought differently about attribution. And some of these conversations about incrementality that were not occurring two and a half years ago are now occurring, and they're curious about it. And I think it depends. It's probably a little bit different for kind of marketers that are in the weeds and then maybe just like, finance, the CFO who's just kind of like, hey, how many customers do we get? What revenue do we get? Like, pretty simply. And the thing is, you know, all of the, all of these models, if they can get to that question or answer rather, and show that improvement, that's what they're happy with.

But I think the first step is asking the question right and getting your finance team to. I've seen this. Marketers are always frustrated by this because the marketers report out on some ROI on the marketing span, and the finance team is like, okay, great, double the marketing spend and we'll just double the revenue number as well. And the marketers are like, oh, wait, no, no, no, no, don't do that, don't do that. It's not going to work. And so, you know, I think it behooves marketers as well as the finance team to have these realistic conversations about what is incrementality, because that's how you're going to be able to come to some realistic budget for next year and realistic expectations about what it's going to be possible to drive with the marketing spend that we're allocating. If you just draw this, like Straight line on Last touch Attribution. You assume that it's all incremental. Like, you're going to get yourself into problems when the board is like, okay, we doubled marketing spend. Why didn't revenue double? And you need to be ready to have, I think, a more sophisticated conversation about that, especially as the world is changing. Cookies are going away, everything is going to. All of your existing measurement methods are going to get less accurate. You really need to be ready to move into this new world where you're not going to have that insight and you're going to need other methods of measurement that are truly measuring incrementality.

Yeah, it's like, yeah, just go ahead and spend all of your money on brand search and retargeting. Like, that will work out well for you.

But I've seen companies, I mean, most companies learn their lesson after they try that one time. But like, I've seen companies be like, yeah, we're going to do it. And then, you know, they face plant. But if you, if you train your ex, if you're a marketer and you're listening to this and you're training your executives to think that way, it's only going to bite you in the long run. You have to be having realistic conversations with them about true incrementality. And that might mean not getting credit for some of the return that you're currently getting credit for. But I think if you take that credit today, it's only going to come back to hurt you tomorrow when that goes away and you don't have another method of proving how much value you're really driving for the business.

Last question or last two questions. Uh, I know that, you know, you're, you're not actually kind of buying media yourself, but maybe you can speak from the, from Harry's or, you know, at Recast. The most, the most efficient spend that you've seen and the most inefficient spend that you've seen. Maybe like a very, like, micro example.

I have no idea.

I mean, I think, like, inefficient.

I've seen a lot of brands get into trouble because they get, you know, they talk to some agency who's like, oh, you need to do brand marketing. And they're like, yeah, we're going to be like Coke. And then they go spend like $20 million on a TV campaign for an ad, a creative piece that they've never tested, never run before, but that, you know, looks really beautiful. And then like, I've seen that fall really flat multiple different times. So that's incredibly inefficient on the efficient side. I mean, look, Facebook is a great channel. Facebook is a really, really good channel. They're very efficient. They are very good at finding potential customers for a lot of different businesses. And I've seen incredible efficiency there and the ability to grow that channel huge amounts. I think for marketers today, what they need to always be thinking about is, what's the next undiscovered channel? The problem with Facebook is that everyone is on Facebook. Impressions are getting more and more expensive. Your efficiency is going to go down over time as more people are bidding on your same audience. And so, you know, I remember when I was at Harry's in 2015, we were one of the first businesses advertising on podcasts. And so we had a great podcast business. Before podcast agencies existed, we had to, like, buy it ourselves. There were not very many podcast agencies back then, but we were able to be very, very efficient because there weren't all of these other companies bidding for those same podcast spots now, like Mercedes and whoever, are all bidding for the same podcast spots. And so it's not nearly as efficient as it used to be. And so the question to every marketer, or at least like CMOs, I think should be, okay, what's going to be that next new channel that we're going to be able to move into, where we're going to be able to get efficiencies before the market bids them away and we're left getting just normal returns like we do on all of our other channels?

Sure. Yeah. And definitely, I think there's an importance there to allocating some budget towards strictly testing and exporting experimentation, too. So this was great. Mike, thank you so much for being on the show.

Absolutely. All right, thanks for having me.

Really appreciate it.

And, yeah, have a great rest of your day.