Avoiding common marketing measurement mistakes | Ben Dutter (Power Digital) | Coffee Breaks

All right, we are, we're at two past the hour here, so that means that it's time to jump in. Welcome everyone to the Marketing Measurement Coffee Break brought to you by Recast. Hopefully everyone's having a great December and is looking forward to a nice, restful break. As a reminder, I promise to you in these sessions is we'll keep this fast paced and informative. The idea is to try to just deliver a lot of really interesting, useful information to y' all as quickly as possible. And two, we will not pitch our respective companies. During the next 30 minutes you will get an email from us with some links to Recast and some information about Power Digital and our LinkedIns. But for the most part we're just going to be really trying to focus on being as helpful and useful as we can. Once we get into the content, we'll start having a conversation. We would love for people to drop questions that they have follow ups places where you want to dig deeper into the chat. So that way we can make sure to keep this as relevant for everyone as possible. Um, so with that, let's jump in. Um, so super excited to welcome Ben Dutter to the, to the, to the coffee break here. Ben is one of the best people to follow on LinkedIn in my opinion. He puts out consistently great content, super thoughtful, super helpful, practical advice for marketers thinking about measurement and strategy and tactics, the whole range really. Um, he has extensive performance marketing, performance marketing experience both on the agency and the brand side and he's now the chief Strategy Officer at Power Digital. So welcome Ben.

Yeah, thanks Michael. Great to be here. And like we were talking before we started, nothing but love and respect for the Recast team. So happy to be here.

Amazing. Well, with that, let's jump in. First question I want to ask you is what are the most common mistakes you see brands making when it comes to marketing effectiveness management or when it comes to marketing effectiveness measurement?

Yeah, I think that they're, I mean, the most common for sure, which is probably no surprise to the folks on this call, is measuring through some sort of deterministic attribution. You know, that's flaw number one. Most common. What I actually think though is more deadly is not looking at the P and L. And I was actually just on a call this morning with a client that is losing money on every customer that they're bringing in because they've never looked at median coordered LTV to CAC and so and factoring in all the costs and getting to an actual margin number and that to Me is step one. If you can't track that and you can't, you know, hold yourself to a profit matrix or whatever your business objectives are, then you shouldn't even care about measurement or marketing yet.

What do you think are the blockers to that? Is it people who just haven't run a business before and so they're not used to looking at the right reports in QuickBooks or like what is driving that?

I think it's the confusion around customer economics versus revenue economics and the distinction between what cohorts of your audience are highly valuable versus more typical. And so for them, their average LTV was skewed way up by a small cohort of super loyal champions who bought five times as much as their normal customer. But when you look at their median or their mode customer, they're actually underwater. So I think it's the data infrastructure to get to that information and then just the analytical chops to separate that apartment. And you know, as a business blended overall they were about break even but they were confused. Like we should be making a lot more money than we are but it's because so many of their customers were low value one and done.

Fascinating. That's a tough problem to deal with. Like I feel like there's a lot of like mobile games and some specific industries that have this like whale thing where it's like their distributions are really skewed by a few super high valuable customers and then figuring out the economics of like how do you get paid back? What are this? It feels really tough to me honestly. Like it's not as easy as if you're or like an E commerce shop with one hero product where it's like it's pretty easy to generalize across the whole range.

Totally. Yeah. It starts to crop up a lot with either subscription based businesses or businesses that have a really diverse SKU set. They have, you know, hundreds or thousands of SKUs. There's going to be a wide range of customer value.

So you and I, I think both sort of advocate that people not overly fixate on in platform marketing reporting metrics. But I'm curious, like what are your thoughts on like when is it a good to look at those metrics or what should you use that for and then what shouldn't you use those in platform reporting metrics for?

Yeah, I'm a big fan of diversified measurement using a kind of a prioritization framework around you know, what data point do I use to make what type of decision and platform metrics to me like let's just stick with meta ads I think is probably the most common or Google Ads, right? Those two for E commerce at least I look at in platform attributed roas and things like that to make intra platform decisions. So if I'm within a situation where I have 10 ads in a prospecting campaign, I will look at the platform results of those ads specifically to determine which ad is winning in this category. Which ad do I think I can scale but I don't affect the overall meta budget. I'm not adding more money to meta because the platform whereas looks good. Right. I look at the P and L, I look at some other sort of holistic measurement methodology like MMM or incrementality testing and that is more of a strategic decision. And so you need a strategic measurement methodology to answer those questions and make those decisions. Whereas a tactical technical decision day to day. I'm an ad buyer optimizing my ads. You can look in that small of a scale and it's likely not to hurt your business overall.

That makes a lot of sense. And you just mentioned that you know this idea of like a framework for, you know, what you look at to answer what type of question. I've seen you, you post a couple of times on LinkedIn about these different frameworks that you have. Can you sort of talk about the like sort of core advice for talking to CMOs, marketing leaders who are trying to figure out how do we pull these different pieces together and what should we look at for answering what type of question?

Yeah, for sure. I think it, a lot of it depends on the maturity of the business. If you're like a seven figure brand, all I really care about is am I growing up and to the right and is my unit economics profitable or within the tolerable ranges of my investment? That's it. I don't need crazy. Mmm. I don't need crazy incrementality test. The likelihood that my media is incremental is very high when you're that small of a business. And so if you're investing and you're not growing, it's probably a fundamental flaw somewhere in your strategy, your business strategy. Either your product doesn't have product market fit or your marketing is really flawed inherently and you don't need a lot of measurement to tell you that. It's like you should be able to feel it as a owner operator. When you start to get more mature though, let's say, you know, in the lower eight little eight mid figures like 25 to 50 million, that's where I start to see a lot of brands Hit a plateau and they're usually they've diversified into three or four channels, they have a healthy amount of repeat customers, they maybe have multiple revenue streams, like they're sold on bricks and clicks and third party retail. And they start to get kind of confused around what is cannibalization, what is synergy, what is incremental, what is not. And so at that maturity stage is when you really want to start doing that validation process. And I follow, I like acronyms and I follow an acronym called beats, which is a priority business experiment, analysis, tracking, surveys. And a lower letter in that acronym should never supersede a decision from a higher letter in that acronym. So business fundamentals, business economics always take priority over everything else. Experiments are next. So that's your incrementality validation analysis, which is like MMM or causal impact or similar data science LED models should take next. Tracking is next and then surveys is cherry on type. An interesting kind of way to collect directional data. Knowing that it's very flawed and biased based on human memory is terrible. So using that framework though, I can very quickly say, hey, I have a business P and L problem, but my tracking looks great. Which one do you prefer? Well, I don't care what.

Fix the business first.

Yeah, I don't care what Meta is telling me about my roas. I care about what my P and L is telling me. Right. So that's sort of the prioritization framework I recommend for clients.

I love that framework, honestly. And I think, you know, we, we advocate for a similar way of thinking about the world at Recast. I'm curious, you know, we talk to a lot of brands that are sort of dipping their toes into experiments and dipping their toes into sort of more sophisticated analysis like MMM or whatever it's called. And there's often, you know, this problem where the business has been so trained to look at like either a last touch roas or first touch roas or even in platform roas that like getting them to shift can be really difficult. Have you experienced that problem? And like, what advice do you have for marketers that are trying to move their organization sort of to the higher levels of this, of this marketing analysis pyramid, let's call it.

Yeah, I, every day I would say that's probably my main job is explaining to CFOs how marketing measurement works and the things that they need to look at. I probably talk to, I don't know, 10 to 15 CMOs a week and just as many CMOs because of that. And it's getting better actually, which is Interesting. I've seen it over the last couple of years, sort of the post iOS 14 fallout has made it more prevalent where people understand terms like incrementality and contribution, not attribution, things like that. But it still is a very recurring topic. And so the advice that I have usually is to prove it out in a small scale experiment and just, you know, try to find a way that you can validate that directionally. The easiest way is, hey, give me a couple of days even or a couple of weeks to turn media off. That is highly quote unquote performant based on your attribution method. Classic example is brand search, right? But brand search has its place. But let's just say CFO says I want to pump more money into brand search and you're as the marketer saying, no, this is not incremental. It's not incremental. Pick a portion of the country and turn brand search off in that portion of the country and monitor total sales, not attributed, but total sales. And chances are it'll have a very negligible negative impact with no brand search. And you could tell the CFO at that point, look, I just saved all this money from brand search in this area. Clearly there's no negative impact downstream, at least in the short term. Let's try it out elsewhere. So having that kind of baby step of let me prove it, let me prove it, rather than let's just trust some other biased data set is an easy way, especially if it's saving money. CFOs want to be profitable usually. So that's an easy sell is like, hey, it's not going to cost you anything. In fact it'll cost you less. And if all of a sudden the, the fear is, well, if we turn off our marketing, our revenue is going to fall through the floor, it's like, well, if that happens, if we turn it off and two days later you're down 25% day over day in revenue, turn it back on. But we got the answer that we wanted. You know, you don't have to be too academic or precious with your experiments when you're at just that early validation stage, when you're mature and everybody's bought into it across the institution and you're talking about incrementality to the board, that's when you really gotta be careful. But just those early days of like waking people up to the reality you don't need to be super academic with it.

I love that. I think, I agree that like, I think sometimes people get too fixated on like, how do we run the perfect experiment? And it's like, you don't actually have to do that. There's a whole range of things that you can do in between nothing and the most perfect, precise experiment you could possibly run that can give you information and signal and like, okay, that can be useful for just making one, optimizing towards a better overall mix. And then two, being able to like make a case and convince people, like, look, if branded search were really super effective, we would have seen a big drop and we didn't, we saw a small drop. And so what does that tell us? And then you can use that to go and make other decisions and run other similar types of experiments, even if they're not like perfect randomized control trials.

Exactly. Yeah. I think there's a, there's been a generation of really deeply analytical people who, and creative people, honestly, who loved to work on the work rather than love working on the mission or the outcome. And I think that people get that confused a lot. And I, even with teams that I work with on a daily basis, I'm like, is this, will this require a data scientist to explain? Then the chances are we're not doing it right. We're getting too in our own head. It needs to be self evident to a layperson of what the results are and that's what the test needs to be. And maybe the test is more aggressive than, or sloppier or there's other noise in the model, but we live in reality and so we have to operate within reality.

I think that makes a lot of sense and it's great advice. Juan in the chat brought up this question about like, okay, you know, if you want to do this turning off marketing in one channel in part of the country, like there is an opportunity cost, right. Potentially you end up losing those sales and depending on the stage of your business, that can be tough. What I often say is like, well one, like you can go the other direction. Like we can take a channel and increase spend in half of the country and see what happens and get a similar analysis. I'm curious. CFOs often do want to save money, but do you ever get that pushback? I'm like, oh man, we're too scared. We can't deviate from the plan. What if we lose sales and then we miss the target that we've promised the board? And how do you have that conversation?

Yeah, you tolerate what is the information that we're willing to say? So I look at it from a percentage of revenue. Let's say that the business is spending 5% of their revenue back on brand search. So that's the total kind of loss of revenue towards profit is 5%. So let's find a market that's less than 5% of revenue for the business and turn brand search off. Just in that market. Worst case scenario, we lose 5% of revenue, but we saved 5% of cost. Fine. And so you just have to kind of do that power analysis of what is the worst case scenario and game it out. And like I said earlier, if all of a sudden you're two days into the experiment and your sales fall off a cliff, turn it back on. I don't want to hurt the business at all, but we got the information that we needed. You don't have to wait 60 days just because your data science person said to. You can clearly see the line dropping if it is incremental. But I can tell you in the 600 plus tests I've done of that nature, maybe 10% show that. So it's usually much more of a slow, gradual decline. And this is where like ad stocking and latency and consideration cycles and all those things come into play. So it's usually a lot less violent than brands suspect it will be. Totally.

I want to change the topic a little bit so you have.