Measuring brand awareness: Search, surveys & more | Elea McDonnell Feit | Coffee Breaks

Foreign I'm Tom Vladek, co founder and co CEO of Recast, the world's most advanced media marketing platform. With me today is Professor Ellie Fate, Associate professor of Marketing and Associate Dean of Research at Drexel University's Lebow College of Business. I got to know her when she was at the Wharton Customer Analytics Initiative. She started her career at General Motors. She's done a stint at Amazon Amazon. She's done a stint at a company called the Modelers and she also is the co author of a book called R for Marketing Research and Analytics. Today we talk about brand awareness, why it's important and how to include it in your marketing mix models. We also talk about branded search and why using it in your media mix model could make sense or could not in some common pitfalls there. We also talk about experimentation. We cover why it's super important to run experiments. The quote I know half my advertising is wasted, I just don't know which half. It's actually more like I know 90% of my marketing is wasted and I just don't know which half experiment can really help you uncover it. It's super valuable. So we talk a little bit about how to run good geolift experiments, some limitations of common practices and then finally we land on, I think one of le's most interesting papers which is test and roll. So if you need to run a quick test, if you've got a hard deadline, how can you do something quickly and really impactfully? So I hope you enjoy this conversation. I certainly did. Really love connecting with Ellie. I think she's a really wonderful person and has a lot of expertise and insight to bring. Thank you very much Ellie, it is great to see you again. Really appreciate you making the time. So we have very much. Ellie, it is great to see you again. Really appreciate you making the time. So we have a number. Your interests and Recast's business intersect in a lot of interesting places. But where I wanted to stun a stint at a company called the Modelers and she also is the co author of a book called Are for Marketing Research and Analytics. Today we talk about brand awareness, why it's important and how to include it in your marketing mix models. We also talk about branded search and why using it in your media mix model could make sense or art Was this question about publicity and this is actually quite a salient topic recently because there was quite a lot of bad publicity in the election. We're not super political so I won't get into it but there is this saying that any publicity is good publicity. Any media is good media. But we've also seen pretty negative business performance. I mean, I think it was about a year ago Bud Light had this thing where a large segment of the US became really mad at them because of some of their ad choices and actually lost the number one spot. So I think it's Modelo now. That's the number one beer. You've done some research showing that some negative publicity can actually be good on brand perception.

Although interesting that you related it to that particular scenario because that might be one where bad publicity is bad.

Yeah.

Let me tell you what we did. It's one of the projects where evidence we were able to bring to bear was kind of the weakest it could be. We were looking at survey data from YouGov Brand Index, which is daily brand tracker, where people say they answer questions like which of the following brands are you aware of? They're presented with the category and click which ones they're aware of. Then they say, which of the following brands have you heard something about in the last week? And they can click a button, red for bad news, green for good news, or gray for no news. Survey one could take, we took that data. They also ask about purchase intent and a few other things we use or.

Could not and some common pitfalls there. We also talk about experimentation. We cover why it's super important to run experiments. The quote, I know half my advertising is wasted. I just don't know which half. It's actually more like I know 90% of my marketing is wasted and I just don't know which half experiment can really help you uncover it. It's super valuable. So we talk a little bit about how to run good juice.

All that data as aggregate measures, daily brand happenings. We found this funny thing in the mobile phone industry. When a brand has a bump in their negative news, later they get a bump in purchase intent. One to two week lag.

Interesting. How did you. Was the negative news sort of common sense or was the negative news something that you saw in the brand index?

Something we saw in the brand index. So we have no idea what it is.

There was some news that maybe you knew about, maybe you did it and then you saw it in the brand index. There are a lot of red buttons getting hit and then later on there's.

Also a lag correlation between green buttons, the positive news and purchase intent. Both seem to be going well. Someone said, show us another category and we did it for the car industry and we didn't find the effect. We're not really sure exactly what's going on, but we think it has to do with what bad news is. So. Oh man, I just spent two hours downloading the new iOS. That's bad news, right? Yeah, but it's not that bad. And it's also something that signals that I still have the phone, I still use the phone. It was worth two hours to do this crazy download. And so we think in the mobile phone category, the kind of bad news you hear is stuff like I broke my phone, et cetera. And in the car industry, the kind of bad news you hear is this car stranded me, I couldn't get to.

Work into a lake or, you know, the battery experiments, some limitations of common practices. And then finally we land on, I think one of Ellie's most interesting papers, which is test and roll. So if you need to run a quick test, if you've got a hard deadline, how can you do something quickly and really impactfully? So I hope you enjoy this conversation. I certainly did. Really love connecting with Ellie. I think she was on fire.

So we actually see a negative effect of bad news in the car industry and we think it's because it's, it's just more important when something bad happens with your car. And also it's worse, like the news is worse. I think that we were never able to actually dig that apart in the data because all we had were these aggregate data correlations.

So one hypothesis that I would have that I'm curious to get your take on is that I bet there's a relationship between the purchase cycle length and whether or not bad publicity is good for purchase. Because we know from the how brands grow research that distribution matters a lot. A lot of purchases are actually super. So if you're just there at the right time, if customers are aware of you or it's in front of you because you have a lot of distribution, you'll grow your sales even if you haven't grown anything else about the brand or made the product any better. It's just a volume thing. My guess is that if you have a lot of bad publicity in a low, fast moving segment, just the fact that your brand is slightly more salient, people may have forgotten why might increase the purchase probability because it's more top of mind. Whereas for a long research cycle product, you know you're going to get into like, oh, why was there bad news? You're going to Google a product and then see that like, oh, the car caught on fire or the GPS sent me into a lake or something like that.

We were Looking at durables with long purchase cycles, the best that bad news can do is take people who are late in the funnel and move them around a little bit. But it can't, you know, it's not bringing new people into the market or anything. There's another paper that I didn't write. It was written by a couple of professors at Northwestern and Cornell that they looked at the, the Trump bean promotion. Do you remember how Trump was promoting Goya beans and his family all promoted Goya beans, as one does. They actually looked at the effect of this on sales. They found it really interesting thing. The boycott was much bigger than the boycott. But then I actually asked them like, well, how, like how many weeks out did you look? Maybe you should look at 12 weeks. The boycott happens really fast and then it's over fast. People supporting, I love Trump, I want to buy the Goya beans and in the next grocery trip they buy an extra can of Goya beans.

But to a really wonderful person and has a lot of expertise and insight to bring. Thank you very much.

See, the boycott you have to experience that counterfactual would have bought the beans.

Right.

Or else you can't see the dip and it just, it's way more spread out because people don't buy beans every week.

And maybe the bicycle is deep, cans.

In the cupboard already. It just takes a while before you see that they have decided.

Ellie, it is great to see you again. Really appreciate you making the time.

So we have not to purchase the beans.

Yeah, that's fascinating.

The paper is called Spilling the Beans, which I think is hysterical.

A number of topics to cover your interests and Recast's business intersect in a lot of interesting places. But where I wanted to start was this question about publicity. And this is actually quite a salient topic recently because there was quite a lot of bad publicity in the election. We're not super political, so I won't get into it. But there is this circle when possible. Current test of my research linked hypothesis would be the McDonald's E. Coli outbreak. I don't know, you may not be aware, but there was some incidentally like it had nothing to do with him. But right after Trump worked at the McDonald's, they came out and said, oh, there's actually an E. Coli outbreak. We're pulling all of our Quarter Pounders off the menu, at least in some regions and it made the news. So I'm curious if in a couple of months, you know, we see like, oh, people have been thinking about McDonald's a lot and they've forgotten why, but now all of a sudden they're hungry for a Big Mac or something like that. They make a purchase. We'll see.

Definitely put E. Coli in that category of way worse than I broke my screen.

Yes, true. Don't want to get to the hospital.

Right.

These kinds of questions are ones that we get all the time.

A lot of our boycott. But then I actually asked them like, well how, like how many weeks out did you look? Maybe you should look at 12 weeks. The boycott happens really fast and then it's over fast. People supporting I love Trump, I want to buy the Goya beans and in the next grocery trip an extra can of Gora beans.

But customers are really interested in parameterizing brand, which I'm sure you know is really difficult. You can measure aided awareness, unaided awareness, purchase intent, but it actually can be pretty difficult to get these metrics right. Brand trackers aren't super expensive. They're super expensive and they're very noisy. It's much more expensive if you want to zoom in on the actual segments that you care about saying that any publicity is good publicity. Any media is good media. So one of the ways our customers want to measure brand awareness or intent is using branded search volume or organic search volume as a proxy. Have you had any experience with that? Is that something that you can speak to?

Yeah, absolutely. I worked on a project with Google. They were really interested in that. I worked on a project where we got into.

But we've also seen pretty negative business performance. I mean I think it was about a year ago Bud Light had this thing where a large segment of the US get really mad at them because of some of their ad choices and actually lost the number one spot. So I think it's modest.

See the boycott you have to experience that counterfactual would have bought the beans, right? Or else you can't see the dip. And it just, it's way more spread out because people don't buy beads every week.

And maybe the bicycle is getting eight.

Cans in the cupboard already. It just takes a while before you see that they have decided individual consumers to agree to answer a survey. That would be like a traditional brand tracker. We asked them about cell phones, cars and retailers. I'm really passionate about durables marketing. I actually grew up in the car industry.

So my first yellow, now that's the number one beer. You've done some research showing that some.

Real grown up job was at GM in the market research group. So I tend to think about durables a lot. Anyway, we had these People say yes, I will be in your study. As part of this study, I will allow Google to connect my search over four month period with a brand tracker in the middle of that period.

Negative publicity can actually be good on brand perception.

Yeah. Not to purchase the beans.

Yeah. That's fascinating.

The paper's called spilling the beans, which I think is hysterical.

One possible current test of my research.

So interesting that you related it to that particular scenario because that might search.

Linked hypothesis would be the McDonald's E. Coli outbreak. I don't know, you may not be aware, but there was some incidentally like it had nothing to do with him.

But so that we could see how they answered the survey and the idea was to relate people's search behavior to their brand attitudes. We had this idea that if people have be one where bad publicity is bad.

Yeah.

Let me tell you what we did. It's one of the projects where have, you know, high awareness, high purchase intent, they're going to be much more likely to actually search for that brand. So to produce that organic branded search. And the thing I love about this project was we had that rich individual level data. So we knew this person answered the survey this way and this is what their search history looked like. That's powerful, right? To have that individual level variation in the data rather than looking at how things are going up and down on average over time. Um, we, we had that data from Google. I never touched the data because data cannot leave Google there. Evidence we were able to bring to bear was kind of the weakest it could be. There was a programmer and an analyst on the Google side that did all of the analysis. I could screen share with them, but I would like write the R code against simulated data and then send it to the analyst. It would break because the real data is not like how you imagined it. What we found out was right after.

Trump worked at the McDonald's, they came out and said, oh there's actually the cola outbreak. We're pulling all of our quarter pounders off the menu, at least in some regions and it made the news. So I'm curious if in a couple of months, you know, we see like, oh, people have been thinking about McDonald's a lot and they've forgotten why, but now all of a sudden they're hungry for a Big Mac or something like.

That, that it's correlated mostly with lower funnel measures like purchase intent or familiarity, not awareness.

But, but not going to necessarily.

It doesn't mean anything. It's not strongly correlated. If I say I'm familiar With Honda or I say format, we were looking at survey data from YouGov Brand Index, which is daily brand tracker where people say they answer questions like which of the following brands are you aware of? They're presented with the category and click which ones they're aware of. And my next car I would be interested in of Honda. Then I'm much more likely to search for Honda in the survey. We also asked them what they own because these are durables. So they own something. Right now that is the biggest predictor of branded search in durables. Do you know what's going on there?

What's going wrong with my car?

Or why would Ellie fight search for iPhone? Because she's trying to do something with the iPhone. Nothing to do with purchase intent but.

With a car I can imagine I.

Turn off this setting.

Exactly. Yeah, yeah. Like my car is making a weird sound or like how do I pop the hood?

Where do I get my car oil change?

We'll see.

Definitely put E. Coli in that category of way worse than I broke my screen.

Yes, true. Don't want to get to the hospital.

Right.

These kinds of questions are ones that we get all the time. A lot of our customers are really interested in parameterizing brand which I'm sure.

So that is the strongest predictor. So if you're going to use the brand trends to data the organic brand search, there's actually a setting in Google trend. They say which of the following brands have you heard something about in the lens where you can say I only want shopping related searches. And I'm not sure exactly what Google does to identify a search as being shopping related. But those are better predictors of brand adages because you need to filter out that half of the search volume is people trying to do things with their current brand.

Right. What you're saying is, is that a lot of search, especially for durables like.

Phones are past week and they can click a button red for bad news, green for good news or gray for no news. Survey one could take we took that data. They also ask about purchase intent and a few other things.

We use cars or complicated machines in some sense. If you're searching for the brand name because you're trying to debug something or troubleshoot something or do something that you have never done before and of course.

You'Re including or buy a case or buy some new tires, the make and.

Model is going to be in the search but it has nothing to do with intent. But what you're saying is okay, Google has some way of trying to Filter that information out and remain with just the shopping searches. But it seems what you're saying though is those shopping searches, at least in principle and to some degree in practice, are pretty sure, you know, is really difficult. You can measure aided awareness, unaided awareness, good indications of lower funnel brand awareness and intent. Right. Purchase intent. But it actually can be pretty difficult to get these metrics right. Brand trackers aren't super expensive. They're super expensive and they're very noisy. It's much more expensive if you want to zoom in on the actual segments that you care about.

So like if we do a causal diagram, there's a lot of problems here because the fact that I own this iPhone also means you're picking up on.

Something probably even more advanced than what I was saying. A lot of our customers are interested in long term brand effects. Like if I do some TV advertising and I've introduced a bunch of all.

That data as aggregate measures, daily brand.

Happenings, people to my brand and I'm getting them from unaware to aware. Is there anything that can tell me that I did that and that will then be later on predicted?

You want to use Google Trends as the predicted, not the predictor.

I think we want to use it as both ideally. Is there some way to connect between like hey, I did a TV advertisement in October and in November I saw that a million more people became aware of the.

Found this funny thing in the mobile phone industry, when a brand has a bump in their negative news, later they get a bump in purchase intent one to two week lag.

Interesting. How did you. Was the negative news sort of common sense or was the brand and then in June, January that million more people that became aware of the brand converted and obviously one of the rate or any of these numbers but you would see that effect in the sales later on because going from October to January is quite a long time. It's how to draw the line. But if you have some intermediate metric, you can start to see the funnel happening in the data. One of the ways our customers want to measure brand awareness or intent is using branded search volume or organic search volume as a proxy. Have you had any experience with that? Is that something that you can speak to?

Yeah, absolutely. I worked on a project with Google.

But it sounds like what you're saying is no, you only see the branded searches sort of right before the sales happened. Right. And as a link in the chain it wouldn't actually add a lot of value, unfortunately. Is that sort of where.

Yeah, because people don't search for brands when they're not shopping.

Right.

This may astound marketers, but people aren't out there just. Well actually my husband is a car guy. He's the exception that proves the rule.

Negative news, something that you saw in the brand index.

Something we. He does search for car brands randomly just because. But it has nothing to do with his purchase. He hasn't bought a new car in the last 20 years. For most people they don't search for Lexus until they're actively shopping for cars. Can you get at someone's deeper attitudes? You're getting this very self selected sample of shoppers.

It's unfortunate.

The best and own shoppers and owners. If you don't clean out the owners.

We'Re going to be disappointing large swaths of the audience here.

I'm sorry but we saw in the brand so we have no idea what it is.

There was some news that maybe you.

Knew are making the YouGovs of the world very happy.

We're making you go very happy. We're making everybody else out. Maybe you did it and then you saw it in the brand index or a lot of red buttons getting hit and then later on there's also a.

Lag correlation between green buttons, the positive news and purchase. They were really interested in that. I worked on a project where we got individual consumers to agree to answer a survey that would be like a traditional brand tracker. We asked them about cell phones, cars and retailers. I'm a really passionate about a little sad.

So does this mean that we are landing then on. Okay, let's say we wanted to do what I was talking about, right. Which is add in some links to the chain like ads predict sales. Right. And we want to add some stuff in the middle in order to build a model that includes things like increasing brand awareness on sales later on. The best measures that we'll have of brand awareness really are these traditional brand tracking things type surveys where you can actually ask people what brands do you know of or do you know this brand? We're asking it directly. That's sort of where we're landing.

I think that's the right answer. And if you're investing oodles of money in television advertising, it's probably worth it.

To do the brand tracker. One of the issues that I've seen.

Both seem to be going well. Someone said show us another cat with brand trackers.

Maybe you could talk a little bit about is that they're very noisy. Right. I mean we talk about like, you know, it's so noisy.

YouGov especially because of the daily data.

Right. It's just like, I mean look, I mean the election is probably one of the most surveyed quantities on earth. You could have one person up plus three and the other person up plus three one day apart.

That's how sampling works. That's why I category and we did it for the car industry and we didn't find the effect. We're not really sure exactly what's going on but we think it has to do with what bad news is. So. Oh man. I just spent two hours downloading the new IO.

I'm hunting for an alternative method where there's actually observed data that we can look at. Knowing that the trackers are this only in your experience are they still valuable? Is there still directional data that you can get out of it?

I have had durables marketing. I actually grew up in the car industry so my first grown up job was at GM in the market research group. So I tend to think about durables a lot. Anyway, we had these people say yes, I will be in your study. As part of this study I will allow Google to connect my had good luck with an aggregate marketing mix model. I've done this for consulting for a pizza company introduced through YouGov. They had the brand tracker and wanted to add a marketing mix model. We had ads predict attitudes and attitudes predict sales. It was pizza, so the purchase cycles are really short. So we weren't like trying to get this idea of people moving down the funnel. But we were seeing strong correlations between basically the different creatives. So this pizza company changed their creatives every two weeks and we could see when the creatives bombed and we could see it faster in a model that wasn't to sales, that was to the breasts. That's bad news, right? Yeah, but it's not brand attitude.

Right.

Where people are just losing interest in this brand. And they had an ad campaign search over a four month period with a brand tracker. In the middle of that period it really was bad.

Really was bad.

Like it actually produced negative changes in the attitude. It was a mea culpa kind of ad. So we used to be bad but now we're better and that that was not effective. They produced negatives so that we could see how they answered the survey. And the idea was to relate people's search behavior to their brand attitudes. We had this idea that if people have, you know, high wariness, high purchase intent, they're going to be much more likely to negative changes in attitude.

I want to switch gears for a bit. You know, we talked a Lot about awareness and publicity.

That bad. And it's also something that signals that I still have the phone, I still use the phone. It was worth it, two hours to do this crazy download. And so we think in the mobile phone category the kind of bad news.

You hear and how to include that in models and I think we landed on it's difficult but doable and you can do it using traditional survey methods.

There are people who've done traditional marketing mix model with brand search as a predictor of sales. It does have positive prediction ability for sales. I've seen people do that in multiple categories in academic papers that have been well vetted. So it is possible that that could be useful signal. But it's not going to get at this thing that you're interested in. It does predict sales, but does it really get us a measure of what people are thinking in the general population?

Yeah. And since you mentioned it, recast includes branded search send which is very correlated with search query volume in our mms.

Actually search for that brand. So to produce that organic branded search. And the thing is stuff like I broke my phone, et cetera. And in the car industry, the caveat.

Though is that we really think of that as a lower funnel activity. And the big thing is to do it right. We believe in it is that you have to include it as both an input and an output. Obviously there is some causal effect of the branded search ad on increasing purchases.

But yeah, people only the kind of bad news you hear is this car stranded me. I couldn't get to work.

Come to search for your branded query. If something else happened, they had to learn about the brand. If you're including it as a predictor alongside all the other channels that you're modeling, it can mediate the effect of tv. TV causes you to learn about a lake or you know, crazy it caught.

On the battery caught on fire. So we actually see a negative effect of bad news in the car industry. I love about this project was we had that rich individual level data. So we knew this person answered the survey this way and this is what their search history looked like. That's powerful, right? To have that individual level the brand.

And then search for the branded keyword and then that causes some increase in sales intent. There's sort of a double counting because did the TV cause the sale or did the branded search query cause the sale? Because the TV caused the branded search query. And so in our model we have multiple stages to account for all of that. We would caution against just including branded search spend or query Volume alongside TV for that reason.

Yeah, they're high.

It can be highly correlated. And if you draw a dag, it's like the branded search then mediates the effect. It stands in the middle. And so if you really want the effect of TV in totality, you have to do something a little bit more complicated.

We think it's because it's just more important when something bad happens with your car. And also complicated.

At least we think switching gears a little bit. I want to talk about one of.

The level variation in the data rather than looking at how things are going up and down on average over time. We we had that data from Google.

You'Re best known for, which is experimentation. You have a lot of great papers on the optimal ways to run A B tests. And it works that you have multiple times you can run a test or just one time and you want to figure out what the sample size should be. But I want to talk a little bit more broadly first, which is why do you think companies are so averse?

It's wor like the news is worse.

I think that.

But we were never able to actually.

Dig that to running experiments. You know, one of the things that we struggle with sometimes with customers and.

Then we pardon the data because all we had were these aggregate data correlations.

So one hypothesis that I would have that I'm curious to get your take on is that I bet there's a relationship between two more broadly is we'll say, hey, according to the model, you should be spending a lot more.

Well, I never touched the data because data cannot leave Google. There was a programmer and an analyst on the Google side that did all the analysis.

I could screen share on YouTube or a lot more on podcasts or a lot less on this channel. Evidence for doing it can be very strong. The payoffs, if we're right, could be very large. Yet sometimes we hit this like organizational inertia. So there's this like risk aversion and is this something that you've experienced and how do you think about that?

I have. I spent a good two or three years with them. But I would like write the R code against simulated data and then send it to the analyst. It would break because the real data is not like how you imagined it. What we found out was as a cheerleader for experiments saying if you want to know the answers to these questions, you really should be running experiments. There's a couple things. One is what economists call an agency problem.

The people deciding the purchase cycle length and whether or not bad public how.

Much to spend on ads have made a career being the person who decides how much to spend on ad. It would be very bad for their career if they were to find out that they'd been spending the wrong amount. And so that makes the decision makers reluctant to find out the real answer because they've convinced everyone around them that.

They know the answer City is good for purchase. Because we know from the how brands grow research that distribution matters a lot. A lot of purchases are actually super. So if you're just there at the right time, if customers are aware of you or it's in front of you because you have a lot of distribution.

That it's correlated mostly with lower funnel measures like purchase intent or familiarity, not awareness. The other issue, I did a sabbatical at Amazon to work on this. With Amazon ads, the execution is actually incredibly complicated, especially in the media space because you need your media provider to randomize the advertising for you. So there's a lot of communication overhead in getting that to happen. And I actually think it's more likely that the platforms will either decide to do this or not because they can build one system and use it for multiple clients. But from a client sitting on the outside, it is tough to execute that randomized experiment properly.

In some channels are much more difficult to experiment than others. Like for example, you'll grow your sales even if you haven't grown anything else about the brand or made the product any better. Tv, the kind of TV that you buy if you make a big national bond. Honda. But not going to necessarily.

That doesn't mean anything. It's not strongly correlated. If I say I'm familiar with Honda or I say for my next car I would be interested in a Honda, then I'm much more likely to search for Honda. In the survey we also asked them what they own because these are durable, so they own something.

It's just a volume thing. My guess is that if you have a lot of bad publicity in a low, fast moving segment, just the fact that your brand is slightly more salient, people may have forgotten why. Might increase the purchase probability because it's more tight on premier channels during premier times in the day, you can only buy that if you're committing a substantial amount of money.

Right?

If you're only committing some amount of money to test it out out, you're going to be on different channels.

Digital remnant, you're on some cooking channel.

At 2:00am Whether or not that's representative of what it would be like if you did a big national bar is debatable. At best. And so that is one right reason.

Right. The platforms have incentives too. The platforms do not necessarily want their customers to know what the advertising is worth. That's part of this whole gestalt of the business. I convince you that my TV show is great and you buy the ads and my TV show right now, that is the biggest predictor of branded search in durables. That's sort of the sense of the business. As things have gotten digital, there's still this deep reluctance to report to advertisers the effectiveness of their ads. One of the things I'm personally sessed with is histograms of advertising effects. So people that somehow amass a large number of experiments or do you know what's going on there?

What's going wrong with my car?

Or why would Ellie fight Search for iPhone really good, high quality mixed model estimates for multiple brands.

Top of mind. Whereas for a long research cycle product.

And there's two of these I've seen. One was with mixed models, but really carefully done mixed models for every product. Product that Nielsen has grocery store data for.

And then you know you're going to get into like, oh, why was there bad news? You're going to Google a product and.

Then see the other one is a set of ads on Yahoo. Ad experiments run on the Yahoo.

You bet. Like, oh, the car caught on fire or the GPS sent me into a lake or something like that.

We were looking at Dahu homepage for a variety of products. When you look at these histograms, what you. With long purchase cycles, the best that bad news can do is is take people who are lazy is they kind of mass up near zero. Most ads don't work, but they have this tremendously long tail. So the adage that, you know, it's attributed to Wanamaker or Lord Lieberhume depending on what continent you're on, that I know that some of my ads work. I just don't know which half. Which ones, which half. Because she's trying to do something with the iPhone. Nothing to do with purchase intent.

Specifically with a car. I can imagine.

Like I turn off the. This setting.

Exactly. Yeah, yeah.

Like my karma is like these histograms actually bear out in the data that that's what's going on. So, you know, like, if I am an ad platform, do I really want to tell half of my customers that they're not getting any value from my ads? Probably not. Nobody really knows what the equilibrium would be if everybody really understood exactly how much their ads were working.

That's going to lead a weird sound or like how do I pop the hood?

Where do I get my car oil changed? So that is the strongest predictor. So if you're going to use the brand trends you to a whole different equilibrium in the ad business and nobody can figure out who the winners and losers would be. So everybody's kind of afraid to go to this measurement driven world even though people like you and me are screaming from the rooftops that this should be a measurement driven world.

I really want to stick with this histogram for a second because I think that there's a couple of different like really insightful things here. So the first thing I'll say is one that I'm just incredibly relieved that that's what the empirical histogram looks like because our priors when we sort of.

Recast in the funnel and move them around a little bit but it can't, you know, it's not bringing new people.

Into the market model is heavily weighted towards zero. It looks almost like an exponential distribution where the mode is at zero and then there's a long tail that had more to do with regularization than it did with knowing what the empirics were. Second thing is I think there that illustrates a very large disconnect between how people talk about advertising and what the.

Reality data, the organic brand search. There's actually a setting in Google trends where you can say I only want shopping related searches. And I'm not sure exactly because most.

People think about advertising as is all being kind of masked around or anything.

There's another paper that I didn't write. It was written by a couple of professors at Northwestern and Cornell that they.

Looked at the, the average effect. Right. If I have a mix of 30 channels I know my paid ROI on average is about 2. I think people are going to think okay, my worst channel is probably like 1.5 and my best channel is probably like 2.5. But they're all pretty close to 2, right?

Exactly what Google does to identify a search as being shopping related. But those are better predictors of target brand outages because you need to when.

In reality it's going to be like 25 of your channels are very close to zero and then 5 of your channels are at like 5x or 6x on average it's 2. Right. But really there's this huge amount of heterogeneity in your mix where some are really performing and most are really not. And I don't think anybody talks about ads that way. You know, out in the market. I think most people talk about ads as all having some sort of like average effect.

Say that all again, but replace channels with creatives.

Totally.

Also, that yes, half of your creatives are probably not doing probably one of the things about these histograms, you hardly ever see a significant negative effect like the two I've seen do filter out. That half of the search volume is people trying to do things with their current brand.

Right. What you're saying is, is that a lot of search, especially for durables like.

Phones or not, have any significant negative effects. They have somewhere the point estimate, like the best fit number is negative. But then when you look at the uncertainty, you're like, yeah, that's somewhere around zero. That's basically all we can tell you. So that's a good thing to know. I should mention the names of papers if people want to chase them. One is a paper done by Anna Tuckman at Northwestern, Brad Shapiro and Gunter Hitch, who are at Chicago. That paper is featured in a Freakonomics podcast. So your audience could track that down.

If they cars or complicated machines in some sense. If you're searching for the brand name because you're trying to debug something or troubleshoot something or do something.

We wanted to hear more. Anna talking about finding out that most ads don't work and what she thinks that means.

I was just going to say we can include all of these in the. In the show notes.

A working the Trumpet bean promotion. Do you remember how Trump was promoting Goya beans and his family all promoted Goya beans on paper by Randall Lewis, who I worked with at Amazon and Garrett Jones. They actually looked at the effect of this on sales. They found it a really interesting thing. The Buycott was much bigger than the Bonson. Who is interested in advertising effects at Boston University? I don't remember the title. It's. It has 273 like that's the number of advertisements they have in their histogram that they measured.

Oh, interesting. I can't wait to take a look. The other thing I wanted to press on this point, coming back to this histogram right there, it's like most things are zero and some are amazing. This really militates towards doing more experimentation. If you can imagine a world where all of your advertising was just the average effect, right. You knew you were indifferent that you have never done before. And of course you're including or buy.

A case or buy some new tires.

The make and model is going to be in the search, but it has nothing to do with intent. I could spend money on podcasts, I could spend money on TV I could spend money on Facebook and we all have a 2x ROI no matter what Or a world where almost nothing is working except for a couple channels or a couple creative are really killing it that pitch to go to a CEO or CMO or a CFO and say.

Look boycott but then I actually asked them like well how like how many weeks out did you in your mix.

You'Ve got some channels that are delivering zero but what you're saying is okay Google has some way of trying to filter that information out and remain with just the shopping searches but it seems what you're saying that owe value to you and some that are really doing amazing we should find out which those are right and the only way to.

Do that can you imagine like it's the difference between mining for gravel and I just need to scoop the gravel out of the ground and it's all gravel. It's all the same stuff that I want versus look maybe you should look at 12 weeks the boycott happens really fast and then it's over fast. People supporting I love Trump I want to buy the Goya beans and in the next grocery trip they buy an extra can of Goya beans. But tis mining for diamonds where some of the stuff is garbage and some of the stuff is diamonds and we're much more in that world where some of these ads are garbage or just useless like the ore around the diamonds and some of it is diamonds. I'm always very optimistic about the advertising industry. I'm really excited to see what happens in the next 25 years. Some of these people who grew up in digital performance marketing that were more of the diamond mining mindset end up.

In the CMO jobs creative seeing a lot of daily data being able to run those shopping searches at least in principle and to some degree in practice are pretty good indications of lower funnel brand awareness and intent right on individual level matchback holdout tests or geotest I.

Want a daily dashboard instead of quarterly.

Deck on my yeah bringing that energy to traditional like if we do a.

Causal diagram there's a lot of problems here because the fact that I own.

This item forms of advertising too. Well I'm hoping that we can see.

More of that energy Phone also means.

You'Re picking up on something probably even more advanced than what I'm happening in the advertising industry. I want to cover one more thing which is okay we've talked see the.

Boycott you have to experience that counterfactual would have bought the beans.

Okay, you need to run experiments, right? Let's give a little bit of color as to how one might run experiments. And one of the things that, that we look at are doing geographic based tests, right. And we've talked like it's not always easy to do this right? With TV it can be pretty difficult, right? You do a national buy that's different from not doing a national buy but with channels like Facebook or Google. It's actually pretty easy right now. So you've done a lot of work on running, right.

Or else you can't see the dip and it just, it's way more spread out.

Because these kinds of tests can you illustrate in the best case how they go and like when some people don't.

Buy beans every week and maybe the bicycle is going to be cans in the cupboard already. It just takes a while before you see that they have decided, you know.

Mines that we all need to watch out for if we're going to be doing geolift or geographic based testing in the future.

Yeah, couple things. Suppose I had two countries that I could advertise in. One easy test is to do a.

Lot of our customers are interested in long term brand effects. Like if I do some TV advertising and I've introduced a bunch of the.

Ads in one country and not in the other, that would be pretty easy to execute. You can just call up your media agency and say I want ads in this country and not that one. That's a normal request. So that has big problem which is.

That those two people to my brand and I'm getting them from unaware to aware. Is there anything that can tell me that I did that and that will then be later on predictable?

You want to use Google trends as the trees aren't really comparable. The central idea of an experiment is that we're going to randomize across a fairly large group of geographic regions or customers. It could be store level if you have a large chain of stores. But we need a fairly large number of these units, at least 100 not to purchase the beans.

Yeah, that's fascinating.

The paper's called spilling the beans which I think is hysterical. And we're going to put 50 in one group and 50 in the other. And because we coin tossed those two groups of 50 counties or DMAs are going to be kind of similar on average. That means you have to do that. And then you're calling your media agency with this really bizarre media plan where I'm going to have ads in Chester county and Delaware county, but not Philadelphia county, not Montgomery county, not Newcastle county in Delaware.

But also one possible current test of my research linked hypothesis would be the McDonald's E. Coli outbreak. I don't know, you may not be aware, but there was some incidentally like it had nothing to do with him.

But Lancaster county. It just gets like your media agency is like what the heck are you doing? And there's like literally someone typing those into your ad buy platform. And they scridicted. Not the predictor.

I think we want to use it as both. Ideally, is there some way to connect between like, hey, I did a TV.

Advertisement and threw them up all the time. Okay, so that's just a weird thing to do. And we haven't built out the machinery for doing that.

In October and in November, I saw that a million more people became aware of the brand. And then in January that million more people that became aware of the brand. Do you mean like Facebook hasn't built it out, Google hasn't built it out? Or those digital channels have it pretty covered, but the legacy media doesn't have it covered. Is this like nobody does this well?

I mean nobody does this well yet.

Nobody does this well yet.

Okay, so there's another thing that the reason my project in Vienna took two doctoral students. One of the doctoral students is working at an ad agency and working with the client. The other one was brought in to actually match the counties. So he has data on the population and the demographics of all the counties in Ireland.

And right after Trump worked at the McDonald's, they came out and said, oh, there's actually to be cool.

And he's matching them together to find pairs that are very similar. And then we randomize within pairs.

Got it.

And that's a lot of work. Why did we do gives you a more precise estimate. If you help things out by finding these pairs, then the groups are going to be even more similar to each other. And that increases what statisticians call power. It makes it so that our estimates of what the lift is get more precise. More precise. So we're not giving you like, oh, the lift is between 50% outbreak.

We're pulling all of our quarter pounders off the menu, at least in some regions. And it made the news. So I'm curious if in a couple months, you know, we see like, oh, people have been thinking about McDonald's and.

150%, we're saying, okay, the lift is.

Like 93 grand converted. And obviously you won't know the rate or any of these numbers, but you would see that effect in the sales later. On because going from October to January is quite a long time. It's hard to draw the line, but it follows a lot. And they've forgotten why, but now all of a sudden they're hungry for a Big Mac or something like that.

Percent over the 193%, if that's what how you like to think about Lyft anyway. So that matching, like you have some.

Intermediate metric, you can start to see the funnel happening in the data. But it sounds like what you're saying is no.

You own Google published algorithms for doing this like 15 years ago. The white paper is still on their website. But there's no tools for how to do this. There's not even R code for how to do it. It's still the Wild west for these geo experiments. So if you guys are building a tool to do that, that's going to be a huge benefit to your clients. The other thing that I think we should be thinking about, that nobody's really thinking about yet, is time. The other thing you can do is what some people are calling a switchback experiment, where in addition to having it on in some counties and off in others, you could do a crossover where you run that for two weeks and then switch them so that now these count.

Make a purchase. We'll see.

Definitely put E. Coli in that category. Counties are on. And that's going to give you even more precision because now you're matched with yourself ideally, although it's not exactly yourself because yourself in the future is not the same as yourself in the past. Right.

And then there's temporal spillover effects because you only see the branded searches sort of right before the sales happened. Right. And as a link in the chain it wouldn't actually add a lot of value, unfortunately. Is that sort of where advertised in week one and you pulled it off in week two. Then there might be some.

There's other fancier designs like we run the ads, then we go dark for two weeks and then. Then we switch them.

Yeah.

So there's lots of options for how to set up the design, but just broad of way worse than I broke my screen.

Yes, true. Don't want to get to the hospital.

People don't search for brands when they're not shopping.

Right, right. These kinds of questions are ones that we get all the time. A lot of our customers are really interested in parameterizing brand which I'm sure.

In addition to geography we should be thinking about time. Time.

Interesting. There's another.

This may astound marketers, but people aren't.

Out There just problem with time though and I think we probably should end here. One problem with time is a lot of businesses are highly seasonal and so and they're not really willing to run the test during the high season. Right. Like I don't think you'll convince most brands that sell high priced items to run a test during Black Friday or Christmas because there's too much risk involved. The persuasion of a brand to do that would be a tough sell. But the other problem is I have.

To tell you, I know a brand that does that.

Oh you do? Can you share it?

Yes, I do. In direct marketing. You know that I have a paper called Test Enroll.

Yes.

And it offers a new approach to planning the size of an experiment. The name came from a marketer you.

Know is really difficult. You could measure aided awareness, unaided awareness.

We were meeting with quarterly just to share notes. We were telling him about the paper. Well actually my husband is a car guy. He's the exception that proves the rule. He does search for car brands randomly just because he's a car geek. But it has nothing to do with and they're telling us about their uplift model and other stuff. And so we tell them about this paper and the head of marketing analytics goes, oh we do that. We do this purchase. He hasn't bought a new car in the last 20 years. For most people they don't search for Lex do that in holiday. We call it test and roll. We just run these really short tests exits until they're actively shopping for cars. Can you get at someone's deeper attitude? If there's really bad ads and really good ads, a very short test will detect that that they run an email campaign on a very small number like a pre test to 500 purchase intent.

But it actually can be pretty difficult to get these metrics right. Brand trackers aren't super expensive. They're super expensive and they're very noisy. It's much more expensive if you want to zoom in on the actual segments that you care about.

500 people. 500 people get message A, 500 get B. Then whatever comes back, they roll. And she credits that to vastly improving their direct market incomes during holiday.

Why?

Because they're getting a quick read. Like it could be that fashion product number one and this is fashion product number two. We don't know which one's going to catch people's attention in an email. If they're the same, it doesn't matter which one we pick. But if one of them's really a rock star, we learn it and then we move and we move fast. We act on that decision and it was motivated this whole idea of test and roll. We've got a roll. Cause it's holiday. We can't.

The holiday is not rolling.

Send it to the analytics team. And we can't think about it for a week. We just gotta go. Testitudes. You're getting this very self selected sample of shoppers unfortunate the best and shoppers and owners. If you don't clean out the owners it's unfortunate. And roll.

Ellie, this was really great. Really appreciate your time. The wisdom that you've shared. I'm sure our audience, our existing customers, you know everybody else that. So one of the ways our customers want to measure brand awareness is interested in the market will be probably going to the show notes and wanting to learn more about the various papers that you brought up. I definitely know a lot of our customers are disappointing large swaths of the audience here.

I'm sorry but we are making the YouGovs of the world very happy.

We're making YouGov very happy. We're making everybody else be very interested in doing something like test and roll during the upcoming holiday. If not for this Black Friday then maybe for Christmas. Certainly for Valentine's Day. The holidays that are coming. Really appreciate you bringing that up. Thank you so much for the time.

Yeah Tom, the feeling is mutual. Every time I talk to you I learn something. It was a delight being here.

Thank you so much. Intent is using branded search volume or organic search volume as a proxy. Have you had any experience with that? Is that something that you can speak to?

Yeah, absolutely. I worked on a project with Google.

That I'll have to have you back. A little sad. So does this mean that we are landing then on. Okay. Let's say we wanted to do what I was talking about. Right. Which is add in some links to the chain like ads predicts sales. Right. And we want to add some stuff in the middle in order to build a model that includes things like increasing.

Brand awareness on they were really interested in that. I worked on a project where we got individual consumers to agree to answer a survey that would be like a traditional brand tracker. We asked them about cell phones, cars and retailers. I'm really passionate about sales.

Later on. The best measures that we'll have of brand awareness really are these traditional brand tracking type surveys where you can actually ask people what brands do you know of or do you know this brand? Where we're asking it directly. That's sort of where we're landing.

I think that's the right answer. And if you're investing oodles of money in television advertising, durables marketing. I actually grew up in the car industry so my first grown up job was at GM in the market research group. So I tend to think about durables a lot. Anyway, we had these people say yes, I will be in your study. As part of this study I will allow Google to connect my advertising.

It's probably worth it to do the brand tracker. One of the issues that I've seen with brand trackers maybe you could talk a little bit about is that they're very noisy, right? I mean we talk about like you.

Know, it's so noisy YouGov especially because of the daily data, right.

It's just like, I mean look, I mean the election is probably one of.

The Most search over 4 month period with a brand tracker in the middle.

Of that period surveyed quantities on earth you could have one person up plus three and the other person up plus three one day apart.

That's how sampling works Tom that's why I so that we could see how they answered the survey. And the idea was to relate people's search behavior to their brand attitudes. We had this idea that if people have have high wariness, high purchase intent they're going to be much more likely to.

I'm hunting for an alternative method where there's actually observed data that we can look at. Knowing that the trackers are this only in your experience are they still valuable? Is there still directional data that you can get out of it?

I have had good luck with an aggregate marketing mix model. I've done this for consulting for a pizza company introduced through YouGov. They had the brand tracker and wanted to add a marketing actually search for that brand. So to produce that organic branded search. And the thing I love about this project was we had that rich individual level data. So mixed model. We had ads predict attitudes and attitudes predict sale. We knew this person answered the survey this way and this is what their search history looked like. That's powerful right? To have that individual level it was pizza. So the purchase cycles are really short so we weren't like level variation in the data rather than looking at how things are going up and down on average over time. We had that data from Google. I never touched the data because data cannot leave Google. They're trying to get this idea of people moving down the funnel. But we were seeing strong correlations between basically the different creatives. So this pizza company changed their creatives every two weeks and we could see when the Creative was a programmer and an analyst on the Google side that did all the analysis. I could screen share. Creatives bombed and we could see it faster in a model that wasn't to sales. That was to the brand attitude.

Right.

Where people are just losing interest in this brand and they had an ad campaign there with them. But I would like write the R code against simulated data and then send it to the analyst. It would break because the real data is not like how you imagined it. What we found out was that it's correlated mostly with lower funnel measures like purchase intent or familiarity, not awareness.

I'm aware of Honda, but I'm not going to necessarily.

That doesn't mean anything. It's not strongly correlated. If I say I'm familiar with Honda or I say for my next car I would be interested in a Honda, then I'm much more likely to search for Honda in the survey. We also asked them what they own because these are durables. So they own something. Right now that is the biggest predictor of branded search in durables. Do you know what's going on there?

What's going wrong with my mar or.

Why would Ellie fight search for iPhone? Because she's trying to do something with iPhone. Nothing to do with purchase intent.

Specifically with a car.

I could imagine like I turn off this setting.

Exactly. Yeah, yeah. Like my car's making a weird sound or like how do I pop the hood?

Where do I get my car oil change? So that is the strongest predictor. So if you're going to use the brand trends. It really was bad.

Really was bad.

Like it actually produced negative changes in the attitude. It was a mea culpa kind of ad. So we used to be bad, but now we're better. And that was not a. It produced negative changes in attitude.

I want to switch gears for that. You know, we talked a lot about awareness and publicity and how to include that in models. And I think we landed on. It's difficult but doable. And you can do it using traditional survey methods.

There are people who've done traditional marketing mix model with brand search as a predictor of sales. It does have positive prediction ability for sales. I've seen people do that in multiple categories in academic papers that have been well vetted. So it is possible that that could be useful signal, but it's not going to get at this thing that you're interested in. It does predict sales, but does it really get us a measure of what people are thinking in the general population?

Yeah. And since you mentioned it, recast includes.

Branded search spend, the organic brand search. There's actually a setting in Google trend.

Which is very correlated with search query volume in our mms. The caveat though is that we really think of that as a lower funnel activity. And the big thing is to do it right. We believe in it is that you have to include it as both an input and an output. Obviously there is some causal effect of the branded search ad on increasing purchases.

But people only where you can say I only want shopping related search ad and I'm not sure exactly what Google does to identify a search as being shopping related. But those are better predictors of target brand adages because you need to filter out that half of the search volume is people trying to do things with their current brand.

Come to search for your branded query. If something else happened, they had to learn about the brand. If you're including it as a predictor alongside all of the other channels that you're modeling, what you're saying is that a lot of search, especially for durables like phones or cars or complicated machines, in some sense, if you're searching for the brand name, it can mediate the effect of tv. TV causes you to learn about because you're trying to debug something or troubleshoot something or do something that you have never done before. And of course you're including or buy.

A case or buy some the brand.

And then search for the branded keyword and then that causes some increase in sales intent. There's sort of a double counting because did the TV cause the sale or did the branded search query cause the sale? Because the TV caused the branded search query. And so in our model we have multiple stages to account for all of that. We would caution against just two tires. A make and model is going to be in the search, but it has nothing to do with intent. But what you're saying is, okay, Google has some way of trying to filter that information out and remain with just the shopping searches. But it seems what you're saying that's including branded search spend or query volume alongside those shopping searches, at least in principle and to some degree in practice, are pretty V for that reason. Yeah, they're high, they can be highly correlated. And if you draw a dag, it's like the branded search spin mediates the effect. It stands in the middle. And so if you really want the effect of TV in totality, you have to do something a little bit more complicated. At least we think switching gears a little bit. I want to talk about one of the think good indications of lower funnel brand awareness and intent. Right.

Like if we do a causal diagram, there's a lot of problems here because the fact that I own this iPhone also means you're picking up on something.

Probably even more advanced than what I was saying. A lot of our customers are interested in long term brand effects. Like if you're best known for, which is experimentation, you have a lot of great papers on the optimal ways to run A B tests. And it works if you have multiple times you can run a test or just one time and you want to figure out what the sample size should be. But I want to talk a little bit more broadly first, which is why do you think companies are so averse.