How to master marketing incrementality and lift testing | Zach Epstein (Haus) | Coffee Breaks

Foreign we are at two minutes past.

So we are going to jump into the content here. Thanks everyone so much for joining us. Welcome to the Marketing Measurement Coffee Break brought to you by Recast. First of all, thanks everyone who is taking a break from Black Friday Cyber Monday preparations to be here with us this morning. We will not pitch ourselves or our respective companies today. You will get an email after this call though with links to learn more about Recast, about House if you'll have any questions you additional questions you can follow up there. Also want to remind everyone to use the chat. So I've got a bunch of questions prepared for Zach. We think it's gonna be a really good conversation but we want to hear from everyone on the call today what questions you have. So if you have a question, if anything that we're talking about raises a follow up, drop it in the chat. We'll try to get to as much as we can and if not we will have follow up content that will address all of the questions that we end up getting. So without further ado, we're going to jump in. With me today is Zach Epstein. He is the co founder and CEO of House and Incrementality Testing platform and expert on testing all across the industry have had a lot of requests for this as a topic at past Marketing measurement coffee breaks that we have. He's a former head of Entertainment analytics and Product Manager at Google and now is full time running House. Zach, thanks for being here with me.

Yeah, really excited to dig in.

Cool. So let's get started. You and I, we're both fans of really aggressive testing for a whole bunch of different reasons. I think I've seen this, and I think you've probably seen it as well. But sometimes marketing teams are hesitant to run more tests or don't run as many tests as maybe you or I would recommend. Why do you think that is?

I think generally speaking, testing is new. It's a new part of the marketing workflow. And I think often people consider the costs of the test and they don't think about the cost of not running the test. Because if you think about what is the cost of inefficiently running media in a way where you don't really know what's happened, you're more reliant on platform reporting or you don't have tools that are guiding you in the right direction, you could be spending a lot of money on things that aren't working. I've seen companies spend millions of dollars on things that they really shouldn't have been. And they go, we added a million dollars in budget in these two weeks. And, and it was like, if they had been a little bit more measured about it, they would have realized, oh, we just kind of lit that money on fire. So I think often there's additional work, there's set up, there's, there's kind of things to manage and then they're like, oh man. And then even as part of that, we might be having a holdout or there might be some opportunity costs, but the value of the insights is really significant. When, when I was at, when I was at Google, I worked really like I was leading an advanced attribution team. I worked really closely with Netflix. One of the ways they described it that I've really, really liked over the years is they described it as a pioneer tax is like, what is the tax of learning like? But we need to understand this in order to be building the right things and to be spending our money in the most efficient way. So I think, I think it's new. I think it's part of a new workflow. That being said, the last thing I'll say is I think it's changing pretty quickly, at least in our experience. Like, I. A lot of the folks we work with are running a lot of tests. I think a lot of companies are starting to build better test and learn cultures. So I think it's something that when you think of like, what the modern marketing toolkit is going to look like this year over the next five or ten years, experimentation is going to just be a critical part of that workflow.

I think that's exactly right. And you know, we both come from like analytics data science backgrounds who are really deeply familiar with like the explore exploit trade off and this idea of like, iterative learning towards maximization. But I think, you know, one thing that I hear from a lot of marketers and like, whether they express it sort of explicitly or not, they have this idea of like, oh, if we run a test and we get bad results, that's a failed test. Right. And I have to, like, I'm always constantly working with them to try to get them to understand. Like, no, if you've got a learning, the learning is valuable. It's not a failed, it's only a failed test if you didn't learn something. Have you seen that or have you figured out good techniques to help people, like educate internal stakeholders about how to, how to talk about that?

Absolutely. I think this is something we come up against all the time and I see it is a good test. Is a test that you think is reliably pointing you in the right direction. It doesn't mean it's confirming the thing that you wanted to see. It doesn't mean that it's saying everything you're doing is great, giving you a pat on the back. There are tools for that there, you know, brand lifts that like, there are always ways of showing things like, oh yeah, we're doing a really good job here. What I think what like a principle that I generally have and especially like, you know, folks like us from analytics backgrounds is test that I believe in myself, test where I'm like, this is really informing what I should be doing and how I should be thinking about it is a good test to me. And it's one of the things that I think is really different from us and where publishers and different companies have different points of view. But some of the publishers, a good test is a test that shows what they're doing is great, that is what they want. They want that result in that direction. Whereas we want to be running experiments or we want companies to be thinking of what helps us become more profitable, what adds more of money to our pockets and you know, brings more to our P and L. And I think a good test is a test we believe in. And a big part of that too is like having the descriptive statistics, having pre test fits, having confidence intervals, having the data to say this is really good. This is pointing us in the right direction and kind of moving us forward.

So that's a really important topic. What are the things that marketers need to evaluate a good test? You mentioned confidence intervals, you mentioned pre test statistics. Maybe walk us through, like what is, what should that checklist look like for people who are starting to get into tech testing and maybe don't have as much of an analytics background as we do?

Yeah, I think a really good place to start is just if you put on your skepticism hat I want to be really skeptical because ultimately I'll say for myself, running a test poorly is probably worse than not running a test at all. Because you're going to feel like you have data points in the conviction that's pointing you a direction that might not be accurate. Whereas something I like, you know, maybe kind of ironic for me as like a heavy data analytics, heavy experimentation person is, I believe people's intuition really matters. Like, like, you know, concept of Bayesian statistics, like we all have beliefs, we've seen things over a number of years, Things are pushing us in directions that make a lot of sense and like that should Be accounted for. It's not something that's wrong. But I think going into the test with general skepticism of how are we getting to these numbers? Do I have the data? Like you know, can we audit this if we needed to audit it. If I was to build a generalized checklist, I would want to know about the pre test fits, I would want to know about the statistical power, how much like, what is the like, what does pre test fit mean?

What does that like, what does that mean?

So in the concept of geo experiments, you know, you can generate a treatment and control split but the big, the big question is are these populations representative of one another? To give you a funny example of a test that I did in the past that did not work out very well. I was working with Disney and we had Orlando and Anaheim in the treatment group and big Disney event screws everything up. There's this volume that we can't account for and it basically made the test mood and meaningless because it wasn't a reliable treatment and control. We probably should have excluded those regions from eligibility for the experiment in aggregate. So what a pre test fit or what some of these tools are is can we track our model to say these geos behave very similarly over a period of time going into the test. Then we make some change. Change could be spending more, it could be spending less, it could be going dark, could be creating multiple cells of the experiment. And then you see those deviations move. So you see this relatively close tracking line and then you see these lines expand. That is the ideal thing. When you don't see these lines expand, that means that the media is probably not as incremental as you were hoping.

Got it. Great. So pretest fit, we're checking, making sure that things are comparable prior to the test and then. Sorry, I interrupted you. What were the other things on your checklist here?

I think statistical power. What is the likelihood of being able to detect lift? Because ultimately probably true for almost all measurement methodologies is it's all about signal to noise ratio. Where is the signal, where is the noise? How do you know that you're spending enough money to be able to observe the difference that you're looking for? So for example, if you know Zach and Michael's wallet business that maybe we'll create next week, you know, with, with like a thousand dollars, we probably could run an experiment, get like first of all, everything there is going to be pretty incremental but with a small amount of spend we'll be able to observe the difference in our historical sales because like there's just not that much data. But if you're Disney, you need to spend a lot of money, like you need to spend a fair amount of money to have the experimental variation to actually observe differences. So I think these are some of the places where having that statistical power is crunching the numbers before the experimented run to make sure you have the confidence of if we spend this amount of money based on our historical sales, based on the historical volume, there's a reasonable likelihood that we'll be able to actually, you know, differentiate the signal from noise and understand the effectiveness of the marketing.

Great explanation. I see people who are sort of new to testing get this wrong a lot. And I think, you know, the root problem is honestly the way that we teach statistics in undergrad. Because people run a test that is underpowered, they get a non significant result, which I'm putting in scare quotes here for people who might be listening in the future and then are like, oh, it means that the channel wasn't effective. But that's not the right interpretation because that test was always underpowered. It was never going to be able to detect a real result. And the idea of statistical significance just means, hey, we couldn't say that it wasn't exactly zero, but that doesn't mean that there's no result there. We can't prove that there's no result, or at least that test isn't powered enough to do that. And so this is a thing that I see as being a real issue is not understanding power and then really not understanding what statistical significance means. And then sort of the difference between just a very uncertain result and a result that actually means, hey, this didn't really have any effect.

Absolutely. I think it's exactly what I said, like what I was thinking before. The dangers of running poorly run tests. It's not worth running a test at all in that context. Use your intuition, kind of use other tools. You want to go into the experiment knowing there's a high degree of confidence, you're going to be able to get the insight that you're looking out. And I think it's one of the situations where the work is all done before the experiment actually launches. Like, you need to have the numbers, you need to have the confidence, you need to have configured things correctly. You need to have representative treatment and control splits, even if you're spending a lot of money, if it's not, you know, inclusive of where historical sales have taken place. Like to give you an example, like to give an example, if you think about, like lawn care brands who have tremendous geographic diversity in, when people are interested in lawn care products in like the Northeast versus the Southwest, you get into really big differences. And, and if you're not accounting for all of these things, like spend is definitely part of the equation, but it's, it's all of these pieces that need to come together to run a really well run experiment.

Yeah, 100%. So we've talked about some of the things that can go wrong in these sort of, in these sort of tests. What are other things that you've seen people get wrong when they're starting to try to run marketing experiments or other common problems that you've seen?

A pretty common problem is people misinterpret the results of an experiment. So I think like number one is people think about incremental return. Like they think it should reflect their CPAs in some regard. Oh, my cost per like my CPA in Google was $10. Wow. My CPA, like incremental CPA is $100. It's kind of not crazy actually. I think that people tend to over over attribute the effectiveness of media. They tend to over like say, hey, assume all of their media is incremental in a way that it's really not. Another really big issue where like I've got a pretty, pretty strong philosophical point is when I was at Google I worked on the conversion lift product for a number of years. Google's conversion lift product is instrumented differently than Meta's conversion lift product, which is instrumented differently than Snap's conversion lift product. And they are not cross comparable, like full stop, different methodologies, different ways of matching individual user records. So I think not having the consistent form of measurement is a really critical one. And then the last piece that I'd probably call out is just interpreting the results in a way that doesn't necessarily make sense. So sometimes when you know, let's say you run an incrementality test and you're like, wow, like meta was really efficient at $100 a day compared to a counterfactual. We're not spending, let's raise the budget to $200. It's like that's not actually what we tested, what we tested as $100 versus zero. So what you could do is you would take what you saw in the treatment, you'd roll that out across the control. That is the expected benefit. But if you wanted to increase your spend, you should run a step function campaign where you have one cell where you're spending more, one cell where you're spending bau one cell where you're holding out and then helping plot out as a diminishing marginal return curve of, you know, what is the marginal CPI between the BAU and that increased spend.

That's obviously a great thing to do.

But then, you know, when I've seen.

When I've tried to do this in the past, sometimes you obviously run into the power problem, right, of like how many different cells do you need to run with like different amounts of spend and are you powered to be able to detect those differences? And so I've always felt it's like it's a really tough balancing act to figure out what's possible with the like any reasonable experimental design and then the power that we're up against to try to combat the signal versus noise problem. So y' all obviously do a lot of like GEO holdouts at house and so y' all are experts in that. And you've talked a little bit about some of the things that can be challenges with that with the lawn care example of where if you just like randomly split the country in half, like you're gonna get bad results because different people like are doing lawn care at different times of the year. I'm curious, like, what are some of the other problems or like challenges when it comes to, to running geographic match market tests or geographic holdout tests? Like what are the, what makes it impossible? Where is it difficult or impossible to run those tests? What are the other challenges that people face when they're trying to design good tests in that framework?

Yeah, I'd say the biggest places, I mean there's going to be certain channels where you just can't have a holdout Influencer marketing often is an example where like something's burned in it is what it is. There's places where I would say, like we work with companies who are kind of like measurement ideological zealots where they're willing to buy TV regionally for a month, understanding they're paying a ridiculous CPM upcharge just to be able to measure that. Most people, it's kind of impractical. Like we have other solutions for time series based estimations. Just hey, where this is like less practical. So really if you can't create a geo control or it's punitively expensive to create a geo control are the examples in most other cases. Like, I'll just say as an aside, when I, when I got into the world of match market testing more than 10 years ago, I saw really dramatic deviations of like, like let's do something in Phoenix and Let's do something in Philadelphia and like let's just look at the difference between the two. You have super high standard error. You anything can happen in an individual market that screws it up and like there's a convention you didn't know about and it largely impacts the efficacy of the results. Also two kind of more analogous to the linear TV example, you pay a huge CPM upcharge to be targeting in an individual place, in an individual time. You need to work at these higher levels of aggregation. So I'd say those are the primary examples. But even I'll say one of the really fascinating things for companies like us, the methods of evaluating experimentation have just gotten so much better. Like we work with companies whose products are only available in four or six states and we're able to get clean reads out of DMAs within those states which would not have been possible five years ago, which is really, it's really exciting to see the innovation on the science side that can go along with this.

Totally. The science side is really fun. I wish I could spend all of my time just on that. We just got a really good question in the chat that I've sort of struggled with. I'm curious like what your perspective on it is. How do you deal with like volatile low volume markets potential or potentially like a skewed revenue profile Right. Where you have whales? Like are there solutions for that? It seems really challenging but you're more up to date on what's possible.

Yeah. So I mean I'll tell you like this is a large part of just like what we as house. One of the reasons why geo experimentation and I'll say just like causal inference in sparse data sets is so difficult is you're trying to understand that we are able to work with very low volume regions like the just the methods have gotten a lot better. When you're generating treatment and control splits like say you're going across the United States, you just make sure you have an equal representation of high volume markets, medium volume markets and low volume markets. So if you think about something like synthetic control as an example, like in an old world where you're saying Phoenix versus Philadelphia, the whether it's like synthetic control or synthetic difference in differences, now you're able to say Phoenix is actually more representative of 40% of Philadelphia, 30% of San Diego and 30% of Portland, Oregon. So you can get this much more prescriptive understanding by creating blends of multiple markets, ostensibly giving you more data points for the analysis. So that is generally true. There are cases where if 98% of your sales are in New York and LA, you're in New York and LA. I'd probably just be doing different levels of spend. But I think generally in that world, I think all marketing measurement wouldn't work particularly well. I think you just run into general challenges and holding out one of those markets is really oppressive. In those cases, I probably would just fluctuate spend. But the last thing I'll say on it is from both an experimentation standpoint and just general marketing measurement, I'm a big believer in just practicality. Run experiments where it makes sense. Don't run experiments where it doesn't make sense. Run the right. It doesn't need to be randomized, controlled trial treatment versus holdout. It can be, we're going to spend 20% more in this market, spend 20% less in this other market. It's the directional insight. But a directional insight can still be really useful.

I think that's totally right. This is a point that I harp on a lot is directional insights can be really important, but you need to know how to communicate those results out so you don't get the underpowered, false signal, not significant result problem. So like, that's the trick is run a lot more experiments. They can be directional, that can be great for driving the business forward. But you need to get. Not get caught in the trap of being like, oh, the result wasn't statistically significant. So it means, you know, it means that that channel wasn't effective, which is just not. It's the absolute wrong lesson to learn from whatever.

Yep.

So, you know, you talked about lawn care and that sort of reminds me of the problem of seasonality. Right. A lot of businesses have this really strong seasonal component. People mow do lawn care in the summer or they buy sunscreen only in the summer, or they only buy snowboarding gear in the winter or whatever. How do you think about testing in that world? Because you're going to be running these tests at some times and they might not apply to other times. How do businesses think about that?

Yeah, what a lot of businesses do is generally speaking, for a lot of companies on the calendar, they have different windows of high seasonality and low seasonality. So they tend to cross compare high seasonality experimentation to other high seasonality experimentation. So to give an example, if you're a sports betting company, the launch of NBA season can be analogous to start of MLB season, even if they're in very different windows of time. Hey, how do people respond to advertising in windows where things are turning up or Turning down for businesses that are, that have snowboarding equipment, maybe like folks are really interested in that between like, call it like October and February. This is where I'd really stress the idea of high velocity experimentation. When I also, when I started working in testing, and especially if you work like through like a Google or like a GOX style test, they're gonna be like, oh, you need to run it for six weeks, you need to run it for eight weeks. And in many cases it's because they're kind of optimizing for their outcomes of being able to no matter what, demonstrate lift. Whereas high velocity 23 week experiments can give you that signal within that shorter window of time. That can give you the ability to kind of derive those insights earlier in the season and then play them out. So I think running more experiments is really frequent. But in terms of how the models work, if you run again really well run experiments, we're taking account of seasonality. So we build that into our treatment and control splits where like, you know, somebody works with us, like we want to look at at least a year of data just so that we can kind of model that into to what we're doing. But the last piece I'll say is like, this is where other measurement tools come in too. Like experiments are part of the portfolio. I think MMs are part of the portfolio. I think platform reporting can be part of the portfolio. It's using these things in sequence where if you see something crazy going on in your platform reporting, as much as I am, generally speaking a skeptic of platform reporting, that can mean something. There's not something insignificant there, like working with MMMs, these pieces should come together and I think be complementary. And I think that's where even being an experimentation person, run high velocity experiments, run really well run tests, but also understand that it is part of the puzzle. It's not the entire puzzle.

Could not agree with that more. High velocity testing, first time I've heard this phrase. I love it. How is, does that just mean, is like, is there anything more to it than just like running more tests faster or like what does an organization need to do in order to really embrace high velocity testing?

Yeah, I mean, just to give like a generalized analogy, one of the ways I describe us, like if you've seen the movie Moneyball or read the book, I like, I liken us to the Jonah Hill character in that movie. We're like, here's the data, here's the, like we are trying to help people make the adjusted like the ROI risk adjusted. Highly probable bets like what is the right two week experiment? What is the right four week experiment? Where can we get away with an 80% statistical power? Where can we get away with a 70% statistical power? What is important enough where we need 90 plus percent. These things operate on a spectrum. It depends the insight you're trying to drive, how much money is on the line, the importance of the network. So where I think about high velocity experimentation is experiments can be used for the different use cases for the business. You can run a lot of experiments, you can run very few experiments and a lot of the folks we work with are running several experiments every month which is like from my state like is awesome. I could never like a couple of years ago I wouldn't believe companies other than the Netflixes and the booking.coms of the world would be doing that. And it's having the information to do it in a ROI plausible way. Doing it in a way where you're able to have concrete enough outcomes like not everything needs to be 95% plus powered. We're absolutely certain like you know the generalized expression of like 50% of marketing is wasted, you don't know which half. And I think I would argue platform reporting has made that even harder because of like conversion optimized signals towards non incremental activities. Like that might be less than 50% at this stage because like people are, it's not random. I think people are actually getting bad information and it's sort of how do we point people in the right direction even if it's not like that dollar is going to drive this amount of spend. Exactly. Because I think like I'm theory certain, I don't think any of these systems are that precise. Like we're all doing the best we can. But if, if you're like I'm going to put a dollar in, I'm going to get a dollar zero seven out. Like it could be a dollar five, it could be a dollar nine. And I think but moving people in that direction is really the effort.

No, I mean I look I love everything that you're saying. This is honestly a thing that we struggle with at recast a lot because I like we obviously understand uncertainty really well. Like an under uncertainty is a fundamental fact of life. Right. Is the thing that we talk about and there's no getting away from it. But a lot of marketers, especially those not trained statistics, they don't, you know uncertainty is generally very unintuitive and a lot of marketers struggle with the concepts of uncertainty if they haven't been trained in it. And so they, they want certainty. And a lot of, unfortunately, a lot of these other tools are built in such a way that like, has trained them to expect certainty. Because you log into Facebook and you see the CPA number and there's no uncertainty associated with that. It's just. It's 75.$4 or whatever. And I think it's. It's. People like us are sort of facing an uphill battle to try to educate people about, like, look, uncertainty is a thing. You can't get away from it, but we can use it to still make good decisions. And we can make decisions, like, really good decisions in the face of uncertainty. But you have to, like, shift your brain a little bit to learn to work in that world. And it's actually the case that the people who are expressing uncertainty to you are the ones who are being more truthful than the ones who are sort of optimizing for some point estimate that isn't actually reflecting that true underlying uncertainty.

I could not agree more. With every single thing you said. All of that uncertainty exists on the ad server side. They just don't expose it to the user. Like, it's all there. It's all in the systems. I worked in the systems. They're just like, but here's the number. Exactly. Like, people just want a number. And I think the way I talk to a lot of people about it, because again, folks like us who just have lived this life and live in a world of uncertainty, just ask yourself in your practical life of like, how certain is any of the things that you do? How certain am I that I'm gonna have lunch today? I don't know. I might. I might not have time tbd. Like, we'll. We'll see how it goes. And it's just. It's really ironic. With huge fundamental shifts in your things that are actually happening in your life, you realize there's some uncertainty. But in your marketing evaluation, you're like, oh, it's that exact number of dollars. And it's like, that's not the way the world works as much as we all might want it to be.

What a great note to end on. So we are out of time. I know a lot of people on this call. Have to get back to Black Friday Cyber Monday planning. Thank you so much to everyone for joining us. Check out House IO, follow both me and Zach on LinkedIn. We're going to be publishing a bunch of content on all of these topics going forward. We got a ton of questions in the chat that we did not have time to get to. We are going to put this into the hopper. We will start producing content that answers all of those questions directly. And then don't forget to sign up for our next marketing measurement coffee break with Ben Dutter from Power Digital. Thank you everyone so much for joining us. For those who are celebrating Thanksgiving, have a great Happy Thanksgiving, safe travels next week and we will hopefully see all of you all again very soon. App that we did not have time to get to, we are.