Amazon & KiwiCo vet Jim Kingsbury on operationalizing MMM for omnichannel brands | Coffee Breaks

Foreign. We're at 903. We only have 30 minutes. And the whole point of this is to keep things very tight and efficient. So we're going to jump in. I think people will be joining over the next couple of minutes as they free up from whatever other calls they are on. So first of all, welcome everyone. This is the the marketing measurement coffee break with Recast. We're really excited to have all of y' all here and especially excited to have Jim Kingsbury here with. Jim is a fractional CMO for large online brands and really all types of brands. He has tons of experience in everything from Zappos to Kiwico to Allbirds. And we're really, really lucky to have him chatting with us here today. Goal for today is to have sort of a good conversation about marketing mix modeling, which is something that Recast does and has been sort of exploding onto the scene and onto people's roadmaps over the last couple of years with the change changes to the ability to track people and as a lot of brands are moving into more of an omnichannel marketing strategy. So we're going to talk a lot about operationalizing MMMs today and really get to rely on Jim's expertise and background in terms of actually having done this at a lot of different brands over the last 20 or 30 years. So thank you so much, Jim, for joining us. Really excited for this conversation. Jumping in. I mean, I'd love to. Well, actually, before that we're going to jump in audience. We would love audience participation. So as people have questions, please drop them in the chat. Marty will be collecting them so that way we can sort of work them into the conversation. So as those questions come up, drop them in the chat or you can send them directly to Marty. And then the goal for this will be to really make it a super informative conversation. So jumping in, Jim, I would love to hear, or I guess this is a thing that I've that I've heard that a lot of marketers are struggling with today is making the transition from like a real digital tracking first measurement style to. To using other types of measurement. And that could be mmm or incrementality and experiments. Can you talk a little bit about how to actually make that happen and how marketers today can think about, you know, making. Helping their organization make that transition?

Yeah, for sure. So I'll start by saying, I mean, you know, things like mmms and incrementality, they've been around, but it's sort of to your point, it's More recently, because of privacy changes that it was previously it was sort of an option, whereas now it's sort of your only choice. And so like you just gotta do it right. And so that's part one, and then part two. You allude to this as well, but call it out is like, you know, hey, all these kind of online, you know, kind of clicky channels, like gray, you know, like if MTA exists, you can track them, but there's a lot of channels out there where there is no click, right? And so, you know, kind of those approaches just didn't work for those either. So to the extent that your mix is evolving to non clicky channels, then that's just another reason you gotta do this stuff, you know, so that's the setup. And if you need to kind of persuade people in your organization, it's just this is the world we live in, you know, like, like we may have like really enjoyed MTA and clickbait stuff, but it's just really, it's not an option now. So that, that's, that's the setup, you know, and then, I mean, so, so, so many lessons. I mean, I think maybe another kind of call out. And again, Michael, you're the expert in this. MMMs maybe have a little bit of a not wonderful reputation historically as being sort of cumbersome and slow and not at all agile, you know, inexpensive and all of that stuff. And, and I get it. And because they, they had mmms, again, correct me if I'm wrong, been like decades and decades and decades. So that's not new, but there's a modern version of mmms, you know, and again, you're at the vanguard of that, doing that, bringing modern MMM to all of us. So then that also helps kind of make the pitch internally, you know, in the organization. So I mean, those are kind of key things. And that's sort of for the entire, the enterprise, all the functional areas who need to be kind of bought in and persuaded. And then marketing teams, it's like, all right, marketing teams and channel managers, you know, you may have been used to all this clickbait stuff and that's fine. And there are cases where you can still kind of use that. But, but the new theme is, and again, many people say this, Michael, you say this many times. You know, it's no, we no longer live in a world of one tool, one way to kind of assess performance and results. You got to get comfortable with looking at multiple tools and signals to help you kind of understand what's going on. And that that's definitely a new skill, but that's a skill we gotta kind of all have. So those are my thoughts there.

Love that. Maybe talk a little bit more about that. Like going from hey, we've got this single source of truth, which is our last touch attribution or multi touc attribution to now we've got to go look at other stuff like that feels like it can be really scary or you know, confusing for other people in the organization. Like how, how do good organizations deal with making that transition?

Yeah, yeah. I mean it is, it is a switch and it is a transition and it is new, you know, so how, how to, how to make sure it goes well. I mean, one thing I'll say, and just to give a concrete example of where we. August, September, about a year ago, a little bit more I did put in place an MMM solution at Kiwico was the company. So we went through this and I'm just going to try to think about how the channel managers reacted to it. The first thing I'll say, and again, Michael, I haven't used your solution but everything I hear is great. But one thing that was true about the solution we had in place was it was actually great UI and great reporting. And so in that sense it actually felt familiar. So in a sense, okay, here's the channel and here's even a campaign or a tactic within the channel. Here's how much was spent and here's what the MMM says that spend return so that, and then you can slice and dice it by kind of time periods and that kind of stuff. So in that sense it felt familiar to channel manager. And so that, that's, that's helpful. So that, that's the, that's the, that's the positive. Like actually it's not so crazy. It's different. But then sort of something else I learned which is share with the group is because this UI was so nicely designed and they're like, oh, super convincing. I would actually say some channel managers almost over relied on the mmm. They're like, oh my God, it's all here in the dashboard. Here's the answer. I, I hit a 3 to 1 last month and it's like, whoa, yes, the MMM says that. But in fact, you know, that's there, there are some limitations and issues and so that's not the full story. And so hey channel manager or whoever, you got to look at multiple sources. And so it's, it's an education process. It's not easy. It's a mistake that I made yeah, so that, that, that's been my experience.

Yeah. I mean, that's super helpful. I mean, maybe how do you coach those marketers on like how to look at those different sources and how to integrate that information together?

Yeah, yeah. Again, I didn't anticipate that issue. I didn't anticipate people being so reliant on the mmm. They were. And so I sort of saw it. And then it's just, then it's trying to explain this is a real need and I wasn't great at it. But just to put it out there, try to explain what are the limitations of an mmm? What can you rely on it for, but where does it really fall short and to sort of go through that education process? And usually an MMM vendor can really help with that process. Maybe the ones very specific thing is, you know, the more granular you get, the less you can rely on it as a really high level rule of thumb. And so that was a really important one. But it's just, it's handholding, it's education, it's not one conversation, you're done. It's just sort of an ongoing exercise, but really important. And the other thing I'll put out to you again to Michael or anyone out there who wants to build this, I think there's a need to build, to create a tool that brings in, okay, here's what the MMM says, here's what the channel says, you know, here maybe if some MTA solution, here's what the solution is. Hey, how Herd survey, What does that say? Put it in one dashboard, you know, so then you're sort of teaching and training people. There it is in one place. And that's how you should think about it.

I love that. Maybe talk a little bit about. So like, you know, you just mentioned a couple of different sources. You've got the platform, you maybe have an mta. You maybe have a How Herd survey, you have an mmm, you maybe have experiments like, can you talk through how you think about what are the benefits and limitations of all of those different methodologies?

Yeah, I mean, they're all fundamentally flawed. They all have their issues. And so again, you just can't rely on one. And it's knowing the strengths and knowing the weaknesses and knowing the limitations to understand what to use. And it's different for different channels. And so I guess I don't have a universal like, hey, hey, here's the answer other than to have that understanding, have that sense, you know, an MM is good for this and really not great for this or how herd survey is good for this but not good for that. And then, you know, your, your human brain is then kind of meshing all of those insights together to come up with a point of view, and I phrase it that way on purpose to say, yes, there is a lot of science, there's a lot of data, but that integrating all of these kind of inputs, that is sort of a synthesis process. That's, there's a lot of judgment for that.

Yeah, that makes a lot of sense. So we just got a really good question that was online or like very much in line with what I was about to ask you, which is you're talking to the cfo. The CFO is trying to help make these budget allocation decisions. They want to know, they want an answer, right? They want an answer that they can plug into the spreadsheet that's going to tell them, hey, here's what the ROI is and that's going to determine budgets. How do you have that conversation with the cfo? What do you actually tell them? Is it you pick one source or how does that conversation, how can marketers and CFOs have a more productive conversation?

I mean, it starts a great question. It starts with the recognition that the CFO is a really head of finance, a really critical stakeholder in this process and that partnership between the marketing leads and the CFO soup. Just understand that it's super important and then it's the same thing is to sort of educate that super important stakeholder into how these tools work, what the limitations are. So that, that's just part of the exercise. I think someone asked about incrementality, you know, with incrementality it's such an important idea and a concept, but it's hard. But things like GEO holdouts especially and then maybe the kind of the platform holdout tools as well. I find that's pretty intuitive. You know, most people can kind of understand that concept, you know, and so for a cfo, it's like, okay, cfo, you know, this, hopefully this is a tool or capability that we have. And then the idea that's really important is you have a model like an mmm, but then you have the, the incrementality studies, the holdout studies and the results of the holdout can inform and give you confidence in what the MMM is. And that's, that's a huge piece that I'll say to everyone is creating that confidence in the, in the model, that organizational confidence, so people trust it is, is a really important part of the journey and making it work.

Absolutely. There's a bunch more questions that I have that I want to ask you on this topic, but I want to change direction slightly to a thing that we talked about when we were prepping for this. So you mentioned earlier in this call, like, MMM sor of has a mixed reputation in the market. I think for a bunch of different reasons. You know, I've talked to a lot of brands that are newer, they tried to implement it, it failed for a bunch of different reasons. There's a lot of, you know, older brands that really rely on it, but other more legacy brands that sort of used it and then rolled off of it because they weren't getting a lot of success. I'd love to hear from you, like what are the most common mistakes or the biggest mistakes you see when people are going to actually try to, to bring MMM media mix modeling into their marketing toolbox at least?

Yeah, yeah. Is it's complex. I mean people like you sort of understand the sort of the math and what's behind it, but your average marketer doesn't. So there's a lot of complexity. So to the marketer it is largely, you know, a black box. How does this work? I don't know, but I'm just going to kind of trust the output. But reasons it fails or reasons it doesn't catch on is that absence of trust. You know, the marketers and other stakeholders, CFOs don't have confidence in what the model is saying, you know, and they're again, a lot of approaches, you know, there's like some open source stuff. I don't really understand the source stuff. That's an option. But boy, if you don't have the confidence, forget about it. Maybe you try to do it internally, you know, at allbirds. We did do it internally, even as apples a long time ago, it was internal. So that, that helping of the internal resource, but even the internal resource was very good at doing the calculations, was not so good at selling it, at helping people understand it, you know, so that, that capability like to sort of boil it down and help people understand and have confidence, like that's, that's another kind of potential failure point. And so then back to sort of you, you modern MMM vendors. I think that's what you're bringing to the table is, is sort of the expertise and the ability to explain this stuff so that Decision makers, marketers, CFOs have the confidence and that's probably the most important thing to get these things to stick.

Yeah, totally. I mean that process of like building up Trust, I think is, it's really hard. I mean in my experience, in my experience doing it. And I think it's like having a good strategy going into the project is super important in terms of who are the stakeholders, what different types of evidence are going to convince them. Right. What convinces the CFO might be different than what convinces the CEO might be different than other things. Casey's got a hand raised. I would love to hear from Casey.

Thanks Michael. I follow you a lot on LinkedIn so I'm really happy to be here. This is my first coffee chat. I thought it'll be my last. I'm, I am attacking this right now and what I have found is that I can only speak in terms of revenue. I don't, I tell every agency partner, retail media partner, partner, whoever I'm working with, I say don't talk to me about impressions, don't talk to me about clicks, don't talk to me about view throughs, don't talk to me about any of that crap. Talk to me in terms of revenue because that's all my stakeholders care about. And if we as marketers can't talk about what we're doing in terms of how we're influenced revenue, then none of the other business stakeholders, especially the non marketing business stakeholders, are going to listen. They're going to walk out of every meeting that we have with them with their eyes glazed over, no idea if our methods are even working and a lot of doubt whether they should give us budget next year. And unfortunately I've been burned too many times at this point. So just like Michael and Jim have mentioned building that trust, I'm peeling all the way back to spend revenue roas and I'm getting their buy in on that one worksheet. And then at the end of that meeting I say okay, next time I'm going to show you how to estimate revenue for things that can't be directly tied to revenue. So in store network TVs direct like things like that where you can't directly tie back to revenue. In store signage. We'll get to that next, like give them a teaser. But right now I feel like so many of those folks at that level, maybe not the cmo, but the other folks are so overwhelmed with all of the vanity metrics that we shove in front of them, they can't make heads or tails of it and then we lose trust. So that's just my two cents. I peeled all the way back like super basic.

I think that's a really good point. And this is another Place where I've seen people have success in terms of doing this. Education is like showing, hey, look, if you take the in platform roas from all of the different channels and you like add it up, it equals two and a half times our total revenue. So like, like that can't possibly be right. And so then you can start to get, you know, your stakeholders to ask the questions of like, okay, well if that's not right, then what is right? And how do we think about making all of those pieces come together and starting from a, like the total amount that we spent and the total amount of revenue and therefore what is the total roas on marketing? And from there, like that's the ground truth, right? And then the goal is to divide that up in a partial and partial out credit in some way. I think is a really, it's a really great way to start having that conversation and to illustrate the need for incrementality and MMM and all of these other ways of thinking about marketing effectiveness.

I'd love to add one point. Another kind of learning is I made this joke at Kiwico. We said, you know, what the MMM model tells us on the first day is as bad and dumb and wrong as it's ever going to be in the sense of it's going to get better. As we do hold out tests as we're doing how herd corroborations and that kind of stuff, it's going to get smarter and smarter and smarter. And so that was a nice way to give people confidence, but also help them understand that it is a journey we're on to get better and better and better.

Yeah, that's a really, really great point and a really like, healthy approach to mmm, I think. And especially, you know, our philosophy at recast, right is like, the more frequently you can refresh, the faster you can get smarter, the more responsive you can be. If you're only doing an MMM project once every six months, you sort of lose that, right? Because it's not going to get you like so much has changed over that course of that six months that the model can't actually get smarter over time. And so being able to do this really frequently I think is helpful for building that, that confidence because stakeholders can see, okay, yes, the results are changing. We change this part of the business, we see this improving. And that like link between the model's results and then what they're seeing on the ground I think can be really, really helpful. Someone asked a question about like practically, you know, all of these different measurement methods, they're all wrong, right. And they're wrong in different ways and therefore there will be conflic. You know, your last tech attribution will say branded search is incredible. Your mmm might say it's terror, might say it's terrible. Like how do you deal with those, like those conflicts in terms of the results practically when you're operating as a marketer in this environment?

Yeah. Again, don't have any simple, simple answers. But you know, I start with just common sense. I mean things like brand searches and last click. I mean, you know, five minutes or three minutes. I mean I think anyone can sort of appreciate why that's, that's really not kind of telling a truthful story or a convincing story. You know, I guess the other kind of thing I was going to share was, I mean I do like anecdotes. Anecdotes are of course by definition just kind of limited examples. But anecdotes can help, you know, people understand, you know, how something is working or not the one I'll share, which I just really like. Again, this is a Kiwico. Kiwico sells a little bit on Amazon and so runs Amazon ads. Right. But again, of course the purpose of the Amazon ads is to drive a transaction within Amazon, of course, like duh, right. And then the me it was in the media mix model and then the media mix model was saying no, no, no, those Amazon ads, they're actually driving revenue. On kiwicode.com, it's like I was like, give me a break, that is bs I don't believe that at all. But then in fact we did the how you heard survey and the how you heard survey I heard about you on from an Amazon ad and they were super close. Right. And so that was a great example because it was this thing that I was personally really dubious of. But then we had these sort of two sources to gain confidence and so it's, it's just an ongoing exercise to build, build, build trust and confidence and understanding.

I love that. We've got a really good question here, Jam, and I'm very curious to get your response to this. How would you explain to an executive who'd never heard of mmm before? And I don't know if you're gonna be able to do it in one sentence, but in the simplest way possible, like what is mmm and what are we doing here?

Yeah, I was hoping you were going to take that one. Yeah, yeah, boy, one sentence. You know, it's a tops down look at all marketing spend to form an opinion of the Efficacy of each source of spend. Something like that. That's not very kind of eloquent, but that's, that's a starting point.

Yeah, yeah, this is a really hard one. And I generally like will give a different explanation depending on who I'm talking to. Like if I'm talking to a finance or a consulting type, I'll sort of give the example of like look, let's imagine that you have a baseline marketing mix and then in some month you like spend a bunch of money on radio that you had never done before. How would you analyze the impact of that radio spend? And you can imagine like the spreadsheet that you would have, that would have, you know, okay, here's our sales radio comes on in April and comes off at the end of April and you could look at the lift in sales that you got from that. And so if you can imagine doing that in a spreadsheet for that radio experiment, then you can start to imagine, well, what if you had a machine learning or a statistical algorithm that could do that for all of your channels all of the time. And so for like finance types and consulting types that like metaphor tends to resonate with them. Otherwise it's I, it's difficult to explain other than by saying like look, it's top down regression. I'm not sure if anyone else has a great, that's a great explanation. If you do have a good one, we would all love to hear it.

My dear friend, chatgpt, professor chatgpt, I want to go ask my dear friend and see what ChatGPT has to say. Good.

That is, please share it with us all once we have an answer. Back on that. Okay, we had a few other questions. Where did we go? So maybe talk a little bit about this idea of using like, you know, you mentioned geolift tests and using that to sort of corroborate mmm. Maybe talk a little bit about that process and the relationship between those, those two different tools for measuring incrementality.

Yeah. So really valuable, really helpful. And I will say during the convincing process, the selling process internally at Kiwico before we did this, that kind of idea, that notion of using holdout tests to corroborate the MMM was really helpful in convincing some of the skeptics who are on the analytics side, mostly on the analytics side. So that's the headline that can help people get comfortable, you know, and then I think I said this before, I'll say it again. Things like GEO holdout tests, pretty intuitive. People can really sort of understand that pretty well and Sort of have a lot of confidence, like yes, I understand that. And so then it's like, okay, we have confidence in GEO holdout test and what it says and then we have this media mix model and then we're going to just compare the two, you know, but, but when push comes to shove, the GEO holdout is the greater source of truth. And so we're going to use that to inform and tweak the Mmm. So that's the good part. So they really, they play really nicely together. What I'll also say though is I love the GEO holdout concept. There are real limitations to that one as well.

I was just about to ask you, why don't we just geotest all of our channels, all of the it away.

Oh man. Because I mean one, you really need a lot of data to need to run for a decent period of time to sort of capture it. But then it basically, it needs to be a big enough splash, you know, you need to be doing something so that, that it's, you can feel it, you can notice it, that the model can pick up on it. And so again, just Kiwico for example, I mean we had like five or six meaningful channels and they were all pretty equal. So in some sense that's good. But then even if you like crank up, spend or turn off spend for one channel, you know, are you going to notice it or just takes a long time to get that signal. And so that, that's the downside. And so I love the concept but there are limitations. And to your point, oh my gosh, shouldn't we just run half a dozen GEO holdouts at any given time? Doesn't work that way. So that's a, that was a learning for me.

Yeah, I think that's exactly right. And there's other channels that are just like, you just can't test really via GEO holdouts because they're not addressable via mar. Like, like podcast is a really good example. Even TV is hard to test in GEO holdouts because national TV is so different from local media that like it's not really an apples to apples comparison. If you run a GEO holdout test, it's unclear to what extent those results even apply to media that's bought nationally just because it's so different. And so there are a lot of these like wrinkles here that can make it really difficult to like really get true. The true, like super high quality tests that you would ideally want. All right y', all, we've got two minutes left. Want to be super respectful of everyone's time. So we're going to wrap up here. What everyone is going to get is we're going to send a follow up email with links to information about Jim and myself and recast and some of the other information that we're doing as well as a link to the next one of these that we're going to do. Thank you so much to everyone for participating. The chat has been on fire with people talking and chatting and it seems like friendships are being made which I love to see. We'll send follow up information. Thank you so much everyone. If anyone has follow up questions for Jim or I, I will provide, we will provide information for how you can get in contact with us. But yeah, thanks so much. Have a great Thursday and have a great weekend. Go ahead Jim.

Thank you Michael. Super valuable. Appreciate it.

Thanks to everyone. Bye y'.

All.

I really appreciate it.