Marketing measurement myths and misconceptions debunked | Prof. dr. Koen Pauwels | Coffee Breaks

Foreign.

All right, we're gonna have more people trickling in over the next couple of minutes, I imagine. But because we like to keep these sessions super information dense and super productive, we are going to jump in. Welcome to the Recast Marketing Measurement coffee break. I am really, really, really excited for, for this week's episode. As I mentioned, we're all, we always try to keep these super, super, super informative and productive. So we're going to have a very tight 30 minutes here and as always, we will not be pitching anything. Just want to be having a really good conversation about marketing science and marketing measurement. And that brings us to our guest today. I'm so excited to welcome Professor Cohen Powells. Professor Powells is a distinguished professor of marketing at Northeastern University. He's published widely in the field of marketing science. I've spent a bunch of time reading a lot of his works both in preparation for this event and before. And he's also super active in LinkedIn. So a really great follow there if you want to be staying on top of the world in marketing science. And so without further ado, let's jump in. Professor Powell, Welcome. Recently on LinkedIn, you shared a commentary from Angus and Deaton on the limitations of experimental results. And this generated some really interesting discussion. I know across the field of causal inference, this particular commentary was a little bit incendiary and there was some really, let's say, dynamic conversation happening on LinkedIn. I'm curious, what is your perspective on what should practitioners take away from this? How should we think about the role of experimentation in actually doing marketing in the real world?

So thanks so much for having me on here, Michael. So I always say, as a practitioner, you work backwards from your decision. So what decision are you really trying to make on the data? And that then influences which data you should use, which methods you should use, and so forth, right? So what I see very often, so I spend about five years with Amazon ads and so they offer AB experiments on a campaign level. So whether a certain campaign, the consumers exposed were more likely to convert to your brand than consumers not exposed. And if that was not Significant at the 95% level, an advertiser would very often say, oh, that means we shouldn't spend on Amazon at all. And we're like, wait a minute, that is not the question at all. That was answered by the experiment. The question answered by the experiment was like, was this particular campaign with its spending level, with its audience and so forth, did that have a significant effect? Consumers exposed versus not exposed? If your question is, should we Spend on Amazon at all compared to our spending on social media tv? That's a different question. If our question is within one publisher, right. Should you spend on sponsored products, sponsored brands or you know, branded an organic search? That's a completely different question. You're going to make a decision on. And different questions and different levels of analysis. This gets a bit academic. They just ask for different methods. So if you want to, you know, really causally new. Whereas, you know, I send an online ad to you, Michael, and I want to know with a lot of certainty whether that made you buy that is just really, really, really hard. Why is that? Because of what we call self selection. Right. So you came to a website and, you know, you revealed yourself to me as an online advertiser as an, as an attractive target because you already had some intention to buy into the category. So experiments try to really cleverly but very expensively figure out a very precise question, right? Is, you know, did Micha causally because of the ad versus not. So most of the questions that I'm dealing with is more, you know, let's call it, for lack of a better word, senior management level, which is, you know, how big should our budget be? What should we spend it on? So If I spend 5% more on, on Google paid search and 5% less on Meta, what is the causal effect on my, on my sales or profits? It's a completely different question. It looks similar. But so things like marketing mix modeling and all of their versions, they in very, in most situations seems to be much better to answer those questions versus that. Michael, as this one consumer, you know, causally got to buy my product just because of the one ad that I posted there.

Okay, I think there's a million things to dig into here. Really fascinating. So you mentioned marketing mixed modeling. It's a thing that, that we do obviously spend a lot of our time thinking about. One of the things that I think a lot about is all the ways marketing mixed modeling can go wrong. Right. I feel like, you know, observational methods are really hard because there's a lot of, you know, especially a complex observational model like a marketing mix model, there's different ways to parameterize it. You can, you know, run lots of different models that have similar fit but give different results. How do you think about, how do you know if you actually got to the right results? Should you validate them with experiments? How do you think about the, the intersection, the interaction between observational methods and experimentation?

So first of all, it's very important and lots of things can and do go wrong with marketing mixed models, right? I mean, my dissertation is a long term version of marketing mix models. I think the first thing that goes wrong right away is there's enough variation in your data. So marketing mix modeling is based on your past data or that of your competitors, right? Or whatever you want to give to a modeler like me. And we're making a decision about the future. So there has to be some kind of trust that your situation, what I call the near past, is going to be similar than the near future. Number two, there has to be enough. Hurry up. So, typical example is suppose that you want me to tell you the effect of your price and advertising because you either can't reduce price or increase advertising. Well, if in the past you have only advertised price promotions. So advertising and price are perfectly correlated with each other, it's going to be completely impossible for a marketing mix model based on the past data to tease these apart. So very often we want to create variation. So I always convince companies to also do experiments. So I call it model experiment. Model experiment, experiment mean because, because the model typically gives enough information and buy in to say, okay, let's set up an experiment which is typically costly and nobody wants to be in the control group that you don't get social media this year and all of that good stuff. So the model gives enough indication that hey, we won't lose our shirt if we try this. And then the experiment creates wonderful variation in the data that we can then model again. So it's, it's very iterative. So I see experiments for instance, as a very cool way to, to validate marketing mix models. There's other way to do that. As you said, there's multiple models you can run. And so I like to just run all reasonable models and then have very objective criteria that like, you know, what fits the data best, what predicts best out of sample, which levels. Can we really do something? The one thing to guard against is that MMMs are just used to get the answer you already knew from the start, right? That they're just thrown in there and that the client just says to you, no, this answer just looks wrong because it goes against my opinion. Rather not a model. We really have to kind of guard against that one. Because I think in the past that has given Eminem a bit of a bad name.

I think you're totally right. We've definitely talked to people who have had that experience or sometimes want that experience like they are looking for someone, you know, it's a cmo. I want to Keep my job. I want you to say how good of a job we're doing and that's. It can be hard. You know, there are economic reasons why you might want to work with that person, obviously, but it isn't the long term good thing for the business. What we found is helpful is exact, is doing what you just described of like model plus experiment. So that way if someone's like, oh, this result doesn't match my intuition or doesn't match what I want to believe, you can say, well, let's go run an experiment. And then that will give us additional evidence at least as to, you know, what's right and what's not.

I would also say I work a lot with financial people. You know, half the time finance brings me in. Marketing department doesn't always like me too much. And so then I also talk a lot to creatives and people who do new stuff. Right. It used to be social media, you know, so forth. And they don't like the bean counters. And I'm like, look, you know, you're a creative and you have this fantastic ad content that is three times more effective than the usual one, but nobody believes you. If you can give the data and let me run my model in all cases, I found something that works extremely well. And if this is also accepted by the people who give you the money, that what makes you shine. If everybody in your organization said, yes, we have settled on the way of measurement and your ad actually was three times better, then you get the kudos and you get the leeway to experiment more in the future. So that's, that's also a key kind of organizational blockage sometimes to run these things.

Yeah, absolutely. I want to go back to testing a little bit. So you know, you talked for a little while about testing and you talked a little bit about like the nuances of testing and how to design a good test that actually answers the question that you have. I'd love if you could talk a little bit about, you know, when you work with these companies, you know, what are some of the things that they do wrong that you have to come in and correct to help them run a better testing practice.

Oh, wow. So I mean there's a huge kind of, this gets more technical. There's a measurement issue in doing good randomized control trials and specifically online with the self selection. I just explained, it's just really hard to get to get correctly. Right. So in the experimental realm, a lot of companies, as I said, the level of analysis, they also like to stop the experiment too. Fast. They're like, hey, we got all the data we need that confirms our opinion. And so very often there's like, surprise and lag effects, as I say, right? So maybe the thousand consumers, customers exposed to your new slogan like it better than the thousand exposed to your old just because it's new. And within a week, they go back to their baseline. And so it was a novelty effect, right? So I typically like to run stuff for two or three weeks to say that's. That's typically the case in my modeling. I typically also, if I have data on that one, I look at what the competition is doing. So I typically have an equation that says, hey, every time in the past you run an ad campaign that was above their radar or price promotion, they retaliated. So if I'm going to predict what your next price promotion is going to do, I better take that into account. And so I think with the Internet, a lot of companies have stopped really measuring their competition because maybe it became too big and I would see the last thing. And this is kind of a pet peeve of me of now reach and frequency, right? So there's a lot of attention now to maximize reach. Just even some friends of mine who says, look, spend all your money on reach, but your full campaign on just exposing as many consumers once. And I'm like, well, there's certain issues with that because this assumption that there's no diminishing returns in reach, which I just don't think is correct. So a lot of people talk about diminishing returns in frequency, right? So if I basically touch you with my ads 12 times versus 20 times, I'm not going to get as much bang for the buck for the last eight times. Right? But in a lot of categories, I need to expose you several times if it's high involvement to even make you consider my brand. And so there's some interesting, kind of interesting peaks there in reach. People haven't talked about it. So if you have a good targeting mechanism, whether it's behavioral or whatever, right? And you expose your first million potential targets to something and you have a certain result, and then you say, no, I'm going to expose the next million, and I'm expecting the same roi, that just won't happen otherwise, your targeting is really bad. So there's also diminishing returns and reach. And I think depending on whether you want to just kind of dominate the market and you don't care much about the roi, are you a smaller brand and you want to have a lot of growth, but at A very reasonable roi and you need to do, do some targeting. So I think that's, that's something that is important to find out. Diminishing returns in reach.

Also, your explanation makes perfect sense to me and we talk about this a lot, which is as you scale up reach, especially with like platforms like Meta and Google that have built in automated targeting and automated bidding, as you get more reach, you are sort of by definition reaching customers who are less of a fit for your product due to the nature of the targeting algorithms. And so this is just like a natural fact of operating in these types, especially in, again, the automated bidding world of Meta, Google, Snapchat, whatever, they all have this property where as you expand reach, you are getting to a less and less primed or worse fit customers. As you go out along that curve.

And you see big business, big businesses do this, right? So they basically put unlimited budgets to their online publishers, but at the target roi. So they say as long as our ROI is above a certain target, just expose as many people as possible because we have the big pockets and so forth. Other companies like to have more of a fixed budget, right? And they want to maximize ROI and they're okay with not growing as fast or, you know, or. And I think that's perfect. So if I read on LinkedIn, for instance, effect like, hey, you know, 95% of B2B customers are just not in the market, you know, currently for your product, right? I'm like, yes, that's right. But then if I say that, then the recommendation, you know, spend your money on the 95% who are not in the market, I'm like, are you kidding me? I mean, Seth Godin always puts it, you should sell nuts to squirrels, right? So, yes, if you have the pockets to expose the whole market, good for you. But if you want to target and you have to kind of focus your resources, then yes, you really want to have those 5% that are currently in the market for your product. And then you also have to build the brand and build the mental availability and be on their shortlist when they're in the market. Sure. But that's another kind of budget and that's another kind of spending principle.

There's totally. Okay, we've been talking about reach and frequency and I actually want to take a step back. So region frequency, I think is, it's sort of easy to understand in digital platforms where like people are identifiable and we know, we know like this user got this many ads and we can sort of calculate it. But the people who I hear Talking about reach and frequency. Most are often people who are working in non addressable media like TV and radio. But I've often always been confused about how they're measuring reach and frequency and how they think about actually operationalizing that. Can you explain to me and the listeners like how does that actually work practically when we're advertising, when we're doing.

TV ads, it is really hard. I was just consulting a company this morning, right. So yes, within each platform and sometimes within also each offline publisher, like TV channels and so forth, they will give you some idea about overlap in customers. Right. So a lot of kind of also now connected tv, they know that I've been watching the show the Bear and I've also been watching Snowfall and the New Shogunau. And so they can tell you as an advertiser, yes, Kun Powell's kind of was an old tree. So if you put an ad there, he saw you three times. So yeah, oh, Shogun is fantastic. It was my love in childhood and I was like, this remake is actually better than I expected. So but kind of, you know, how do you get cross channel reach at frequency and overlap? That is, you know, that is really work in process. So if there's anything that associations can do to help us there. I would put one caveat though. I mean, and I don't think, you know, marketing is always excited. It's not never 100% perfectly measurable. And I'm okay with that one. I think there's kind of a lot of companies that I work with, you know, big CPG companies, I think they overestimate consumer annoyance with ads. I just haven't seen it in my research. I mean, the one point where you can really annoy consumers is emails. If I get three or four emails a day from your company, I'm getting annoyed even if I'm a loyal customer. Right. Whereas if you look at people being exposed 6 times, 8 times, 12 times, 15 times to your ads, in much of my research I didn't see any reduction in likelihood to buy. So there's just very little evidence. So some companies, for instance, put caps at three exposures. So they want to measure it. Exactly. And they want to only expose you three times per week to their ads. And I'm like, I think that's a bit too conservative. And of course one of the reason is that they're so uncertain about this overlap. Right. If they would get better measurements, then they could better plan for it and they could control it better.

So just to summarize that point, the idea is that there might be diminishing marginal returns to these additional touches, but it's not going to go negative. You're not, at least in your research, haven't seen it where like, oh, I now hate this brand because I've been advertised too much. It's just not a thing. Is that, is that a good summary?

I haven't seen it and I have one specific research about that. Right. This was a toilet paper brand that had a very big national campaign and you saw that some people were getting annoyed. Why was that? They started unliking the brand on Facebook and that's a pretty big step that you don't do that. Right. Same thing for your email newsletter. So people may unsubscribe from your newsletter, so very often they show it to you and at least online you can actually track that. And so then it's really time to say, oh, are we overdoing it? So this goes to. I think you had a prep question about long term, right? So even if you can measure things in the long term, I'm an engineer, so I think about input, throughput and output. And my fascination has always been what are the intermediate metrics in your throughput? So what can you tell from consumer behavior that happens way before they like stop buying from you or complain about you, but that, that's an indication that, you know, you're either doing good stuff. Right. Because more people are, you know, asking information about your brand, telling about it, or that you have overdone it and people are unsubscribing and so forth.

Makes a lot of sense. How do you think about the role of survey work in measuring marketing effectiveness and like brand awareness? So I mean, we've talked about experiments, we've talked about media mix modeling. Surveys are another thing that happen that I think some people really like and put a lot of faith into and other people don't do at all. What do you think is the appropriate role of surveying in terms of understanding marketing effectiveness and what we should be doing strategically?

So this is absolutely great. Right? So negative of service. Right. I think oval gives that, you know, people don't do as they say and they, they don't say what they think. So there are very known biases in service. So you typically catch people sometimes online you can catch them while they're purchasing in your ticket catego, but typically you catch them off guard when they're not really thinking about you. Specifically, if it's a low involvement purchase, they're likely just make up stuff to answer to the survey instead of very kind of deeply thinking about how do I make purchases, do I really like your brand or love your brand and all that stuff. But I think the pendulum has swung a bit too much against surveys. So people have much more behavioralist approaches, right? And the Ehenberg Boss Institute is a wonderful example. They're like, well, surveys can of are useless because brand attitude just change after behavior. So why do I like Coke over Pepsi in a survey? It's just because I've been buying Coke for 10 years and not Pepsi for 10 years. So I want to make my answers consistent with my behavior. And I said to you 100 survey questions, Coke is fantastic and Pepsi sucks. So so in contrast I have found in my research lots of instances where the brand attitude changed first and then the sales change. So this whole kind of old marketing thinking that hey, to really grow as your brand, you have to get more people aware of your consider it, like it, talk about it to their friends. That's still true. And surveys get to why people do something and I think that's absolutely important. So I have a paper out that's freely available that I can share with your audience that looks for 36 brands, all kind of different categories from cars to like superior. So we have for these brands kind of both survey metrics and everything people do online and the typical clicks and visits, right? And so what we find is the that to explain your short term sales, the online metrics are much more powerful. But to predict your brand three months out, the surveys were more important. So for instance, if somebody visits your website, you know, you're jumping up and down, people are ready to buy. No, in a lot of cases they got there by mistake. For instance, Microsoft figured out that 55% of people already had their product, were not in the market, but had questions about it. Sometimes I go to a website because I have decided against your brand, but my wife absolutely wants to buy you so I have to get information to use against her. So there's lots of reasons why people do what they do. And if you only have their behavior, what they click on, then you miss that. And so I'm now working with several brands that, that are really kind of have only done behavioral, you know, research for 10 years and they have now restarted surveys. And so, and so one of their questions is, hey, in which cases do surveys gives us information that the more behavioral stuff can't, right? When do they change weeks before people's behaviors so we can actually try to preempt that and take action. And I think surveys are still Very useful in that regard.

Wait, so I would love to hear some examples of what are the actual survey questions that someone might ask that are really going to give this insight that you think is really valuable.

Yeah. So I'll give you one example. This is like 25 years ago when I had a Coke presentation. So these people probably long left the company. I can talk about it. So they were very proud. So they presented, and I was close to 100 questions, right? So I think about, you know, 80, 90 questions that they asked in representative samples in every country in the world on regular take. So you can imagine how expensive. And there was a whole bunch of stuff, and I stopped them. And I'm like, every single One of your 85 questions is, how fantastic is Coke? Is it only moderately fantastic? Or it's. You don't ask one negative question. Why not? And they're like, well, like what? And then so two years later, they hire me because they're like, hey, this is interesting. We don't have enough market share in, let's say, Greece. And why is that? They did focus groups. You know, Greek mothers think that Coke is not very nutritional, so they don't want to give it to their kids. And I'm like, hey, you know, that would be an interesting thing to monitor. So I always think when I'm consulting a company and making dashboards for them, like, I'm like, okay, to be successful, what is all the things that need to happen and what are all the reasonably likely failures? You know, what could go wrong? Right. So let's say we have a model that says, hey, you should increase your price by 20% or you should cut your advertising by 50%. Well, what does go wrong? It's not just consumers. It's also maybe your retailers you distribute won't carry you anymore if you stop advertising. Right. If you increase your price a lot, it may be that they really like you and give you prominent shelf space and so forth. So think of it outside of the box. So if I pull these levers that I have, what can go really well? What can go really wrong? And then in surveys, can we ask consumers, but also, like retailers and so forth, you know, what do they really care about? And get to that one. I'll give you a quick example. This is a beer brand that I worked with. And so they were tracking sales and everything was going fantastically. And they also tracked the survey, and they asked people whether they were in love with their brand, engaged or married to their brand. And so for some reason, they had married to as the highest level of their scale, and they had a big dip in that one. And so they were all in panic because they were also paid based on that metric, by the way, if you put compensation based on that one. And I'm like, well, this is really interesting, but can we kind of analyze if that's really a leading indicator of your sale? Because maybe it's not. A very quick other example, this was a car brand, right? And so they were very happy because I think they only have 1 or 2% market share in the country, but they were the number one car Facebook brands, and they had engagement and everything out of the wazoo on Meta and absolutely no move in their sales. So I'm like, well, you know, you run a car business. I don't what is absolutely needed to get people in your new cars. They're like, oh, that's easy. Test track drives. And I'm like, well, have you checked the correlation of your engagement metrics with test drives? We did, and it was absolutely zero. And then you can say, what's going wrong? Right? And then we asked people or like, well, are you actually in the market for a car? And then if you are, are we putting the right kind of content on our Facebook page to make you interested in going for a test drive? But I think these kind of basic questions are very important.

I love this. That's super actionable. It sounds like if you're going to be doing this sort of survey work, the first step is like, you need to make sure that the thing that you're asking about is actually, actually relevant to the goals you have as a business. Because if you're not, you're just going to get distracted by people answering survey questions, even if it doesn't matter for sales and the things that we actually care about. Okay, last question I have before we're going to wrap up. What's the number one myth that you see practitioners believe about marketing that is contradicted by the research or the science that you actually do?

I would say, number one, everything is different this year. I mean, you know, so I joined, you know, the US in the end of the 90s, where everything was going to be so different on the Internet. Brands were not going to matter. It's frictionless commerce, competition, just a price click away. And brands matter more online than offline. So I think kind of the continuity that you very often get from experienced consultants or professors that have seen things before, every year there's so much old wine and new bags that I find kind of problematic. The second thing Is that so? I think a lot of managers, they are of two camps. They either say, oh, you didn't do a case study in exactly my industry, so it's not relevant to me. So learn also from other industries are the opposite. Say, oh, somebody like ebay has figured out that Google paid search doesn't work for them at all. So paid search, is that the complete generalization? Of course there's one study out there and there's all of this kind of things. So that means that I shouldn't do paid search. And I'm like, no, check for your brand, for your goals. You know, run a marketing mixed model, run an experiment and just see does it work for you? Because it's not, because everybody's going in direction X that you can profit a lot by going in direction Y. I.

Think that is exactly right. I love it. That is a great note for us to end on. Everyone, thank you so much for attending. As always, a couple of calls to action here. Please follow myself, Michael Kaminsky and our guest Professor Cohen Powells on LinkedIn. You can check out his research@marketing and metrics.com Ton of great resources there. A lot of his papers, they're wonderful. And then keep an eye on your email. We are going to send a follow up from this meeting with links for you to click on, things for you to check out and also keep an eye on your email. For our next coffee break in April, we've got some great guests. We're just finalizing dates and doing some coordination but that will be coming out soon. Professor Powells, thank you so much. Have a great weekend. Everyone else, thanks as always for joining. Appreciate the time. Have a good one.