Why most brands fail at incrementality testing | Andrew Covato (Ex-Snap & Meta) | Coffee Breaks

Once you run a test, you have all of these problems, but even before that, like getting up the organizational courage to actually do it. I'm curious if you have any insight into that.

There are unfortunately a lot of organizations that suffer from the lack of ability to get started and a lot of it comes down to culture. There's a real aversion to risk, aversion to failure. If folks are micromanaging kind of day to day or week to week changes in some kind of growth KPI to a level that goes beyond just, you know, what would make sense from a normal business perspective, that then you've got this serious allergy really to trying something that may or may not work.

Andrew Cavado is the founder of Growth by Science, a marketing consultancy focused on driving growth through data driven experimentation, measurement and advanced analytics. With a background that spans leadership roles at Google, Netflix, Snap and Meta, Andrew brings deep expertise in growth strategy, causal inference and marketing science. In today's session, Andrew and I talked about the cultural problems that will stop an organization from experimentation and how to change that. The explicit and implicit measurement tax Being data driven but not falling for analysis paralysis why Andrew doesn't believe in brand advertising when brands should start thinking about incrementality, the question is last click a waste of money. You should definitely dive into that and much more. We'll dive right in and hope you enjoy it. So Andrew, thanks for joining me. It's so good to have you. One of your taglines for your business and shows up on your LinkedIn profile is test, Learn, Grow, which I love and I talk to a lot of people about this and I think everyone, at least today in the year 2025, says that they want to test and learn in their marketing program, but many teams struggle to actually live up to that.

Why do you think that is a great question. Really great to be here talking with you, Michael. Big fan of everything that Recast is doing and so just wanted to throw that in there before we get into this question. But yeah, in terms of why folks struggle to test, I think we have to understand, well, what does a test mean actually? And you know, our company is growth by science with an underscore on science. And so everything we approach is with a scientific mindset using the scientific method, which is to create a hypothesis, to test that hypothesis, analyze the results, and then iterate and come to a conclusion based on the analysis that you've done. But when we use the word test, it has a specific connotation in the field of science, especially if you think of Something like clinical trial, which is the most classic test that I could think of. You're creating a statistical environment where you're able to understand whether or not the treatment that you are exposing, you know, a certain group to, if that's having some kind of effect on, on a KPI or outcome that you're, you know, that's important to you. And I think that a lot of marketers, and this is changing, I will say definitely over the last two years we've seen a definite change in this perspective. But I think a lot of marketers historically have not used test in that particular way and they've not considered it to be that way. It's more of like, hey, let's try something and see what happens. And while in some that can be valid, let's try a new channel or let's, you know, try a different, you know, creative or something like that, it's not a true test in that you're not creating an environment where you can understand the causal impact, specifically that underscore causal impact, you know, of, of the thing that you're, that you're trying to change or that you're trying to throw out there. So I think that's probably the first challenge. And then I would say the second challenge is interpreting the results. So because these tests are statistical in nature, they often, you know, they always have some sort of uncertainty that's associated with them. Marketers have been trained to have extremely precise, but I would argue extremely inaccurate data that they've been using. You know, specifically, you think of something like last click, it's extremely precise. You've got a very specific number. But it's often wrong in terms of describing the incremental impact or the causal impact of something that you know, of a channel or treatment of some sort. So, so it's that it's making that leap from having a deterministic figure to a more probabilistic figure and distributions and interpreting confidence intervals and all of that. That, that I think also causes marketers to sort of get hamstrung and, and to not be able to action on true test results.

So I totally agree with that assessment. And I see all of those same problems in our practice as well, of course, but even more fundamental than that, I often see organizations not even running the test. They don't even get started. And I'm curious if you have any insight as to what causes that we see testing, especially in the field of product analytics. That's very ingrained. Every product manager is expected to run a lot of tests and think really hard about it. But maybe we see less of it in the field of marketing. I'm curious if you have any insight as to why that might be, why, you know, even, you know, once you run a test, you have all of these problems, but even before that, like getting up the organizational courage to actually do it. I'm curious if you have any insight into that.

Yeah, yeah. You know, I think there are unfortunately a lot of organizations that suffer from, from, you know, the lack of ability to get started. And a lot of it comes down to culture. At least in our experience, what we've seen. If there's a really aversion to risk, aversion to failure, if folks are micromanaging kind of day to day or week to week changes in some kind of growth KPI to a level that goes beyond just what would make sense from a normal business perspective, then you've got this serious allergy, really, to trying something that may or may not work. And so I think that businesses will. Not to say that it's impossible to grow a business without ever taking any risks, but I would say you're unnecessarily stacking the decks against you. And, you know, I think a lot of the most brilliant discoveries that have happened in humankind have happened because people have taken risks and because people have done something that maybe has failed a few times before actually being successful. Right. And if you take that and apply it to something a lot more constrained like marketing, then, you know, I think there's a very valid case to be made for having a culture of experimentation being okay with, with failures, with some things trying to work, but also creating an environment where the failures are not existential.

Right.

You, you don't put all your eggs in one basket. You've got to understand how to test, how to put guardrails around a test so that it doesn't impact your, your business as a whole. But, you know, be okay with, you know, forgoing some kind of growth, having some kind of opportunity cost that, that, that exists when you're making an experiment. And, you know, this is something that I've spoken about a lot. There's something I like to call a measurement tax. Okay, measurement tax. What is that? You know, there's an explicit or an implicit measurement tax. And I think marketers are a lot more adverse to the explicit measurement tax, which is, hey, I'm going to, for example, hold back some ads from certain geographies. I know that by doing that I will have a slower growth rate in those geographies. And I can estimate that it's going to be within some range. And that scares me that I'm not going to have those conversions in those geographies. And that's the explicit measurement tax. But, you know, if you don't, if you don't, you know, run that experiment or run that test with that, that measurement tax, you have an implicit measurement tax, which is. I am running some kind of measurement model. It's not calibrated. I haven't tested something. I know that it's probably not working in an optimal fashion and that I'm paying for some conversions I would have gotten anyway, but I don't know what that is. But because everything's okay and the business isn't crashing, I'm going to just stick with it and I'm not going to bother, you know, if it ain't broke, don't fix it kind of deal. And that, I think is very dangerous. And we find many, many times, I would say almost 100% of the time when we do an explicit test that the explicit tax is far less than what the implicit tax was.

So how do you help organizations or people make that jump? I think we've got a lot of, like, listeners or viewers who are, you know, they're maybe senior individual contributors. They're maybe like director level, and they are facing friction, you know, up the chain about like, oh, whether it's communicating uncertainty or, you know, trying to propose this experiment. And like, the people at the top are very focused on the explicit part of the measurement tax. Like, what would you recommend to them in terms of setting this up or communicating these ideas to help these organizations take that first step or the first couple of steps.

Yeah, I think it's really outlining all of the possible scenarios. Here is the worst case of what may happen if that opportunity cost, really highlighting what is that opportunity cost of doing a holdout experiment, turning off ads in some GEOs, what is the worst possible thing that can happen? And then what is the best possible thing that can happen? If, you know, let's say we turn it off and things don't, don't decline as much as we, as we expected, which actually I would argue is a really bad thing because then, you know, your, your ads are not working as, as hard as you think they're working. So, but, but, but I think it's, it's, it's understanding, like really quantifying that explicit tax, that explicit measurement tax, but also quantifying what could the outcome be if, you know, we really understood what that implicit Tax was. And you project that over long periods of time, you can see that there's actually potentially a lot of savings.

Right.

There's a, it's a double edged sword though because especially for a marketing org that's been running, you know, reasonably well or perceived, you know, they perceived to be have been running reasonably well and you all of a sudden saying, hey, actually we're 10% less efficient or 20% less efficient than we thought we've been, you know, that kind of calls into question, well, what have you guys been doing? How did you not catch this earlier? And so there is that double edged sword. And so again, it really goes back to the culture. I would say that unfortunately if there isn't a really good culture of experimentation, executive team that encourages folks to dive into things and celebrates evolution, even if that means, hey, something we've been doing, the status quo is not right, then I think it's just that quantification. But without that, I think any organization will really, really struggle to make meaningful progress. And I would argue that I personally wouldn't want to work for an organization like that if I didn't feel that I could help it evolve beyond that. And we haven't met a lot of folks that are that way inclined to be honest. I mean, oftentimes when you show smart folks what you could get and what's the worst thing that could happen, even if it's scary, I think people do tend to lean into that. And so if you've got reasonably smart executives that have a modicum of logic in their approach and philosophy, then that quantification exercise I think will work out. And if they don't, frankly, you should rethink if that's the right spot for you. Right. Is there a good longevity in a company whose executive team doesn't have those things?

Right.

I think that's an amazing answer. And so just to summarize, I think that the key part there that's really insightful and that I think listeners can take away if they're trying to go down this route, is really trying to quantify and make explicit both sides of the tax, as you say, so say like, look, if it goes this way, this is the worst that could happen. If it goes this way, this is the best that could happen. If we continue on our path and we're actually wrong, it's that implies that it's actually costing us this amount and really try to make that like much more tangible for the executive team to get them to again, not just focus on the explicit part of the tax, which is easy to focus on if you're a busy executive, but misses a lot of the story. And so I think data scientists, marketing analysts, that's where they can really help the organization along just by putting, you know, apples to apples comparison for these different values, some of which, you know, aren't necessarily obvious when you're first looking at this idea of running an experiment or turning some channel off or whatever it might be.

Yeah, I think that's a great summary. 100%. Yeah.

On the other side, you know, I work with a lot of organizations that are very data driven and they take that as a point of pride. I imagine you do as well. One of the problems with that I have seen is that sometimes they can sort of go too far to the other side and they can descend into analysis paralysis where they like can't make any decision without, you know, a ton of data behind it. And that can cause, you know, the sort of the gears of forward progress to really slow down. How do you think about helping companies be data driven but avoid the sort of downside of analysis paralysis?

Yeah, that's, that's a really great question. I think this comes down to prioritizing the highest leverage macro questions that need to be answered by an org and, you know, taking this back to marketing, one of the most important tests that I think every marketing team, and I've said this, you know, in a number of podcasts and I think in other forums as well, if you were to ask just about any cmo, what is the incrementality of your overall marketing spend? I find you get one of two answers. You know, one that's like, oh, I haven't really thought about that. I mean, it's, you know, they're kind of like shocked at that question. Like, oh my gosh, it's like it's such a basic question. I haven't really thought about it. Or some kind of really long kind of explanation. Oh, it's this and that. But it's not like an explicit, yes, we've tested it and we know what it is within some air bounds. So that is typically one of the tests that we recommend running and we help our advertisers run. And it's a lot harder than it seems on the surface because you've got, for so many technical reasons, right. There's channels that cannot be turned off at a GEO level. There's, there's confounders, this thing, channels that take time to wind down and wind back up. We can go into that and probably spend Hours discussing why it's challenging. But net, net, the closer you can get to answering that question, the, the, the, the more you've got a framework and a baseline from which you can, you can design the rest of your learning agenda and design what other tests you need to run. I think a lot of marketers, you know, especially the ones that are very data driven, kind of jump to this, like, well, let's just test everything. Okay, we're going to increase our budget by 3% on this Chann 1% of our total budget. Well, let's design a test around that. Okay. Blue or red text in this creative. Let's design a test about that. And then you end up suffering from, you know, it's impossible to measure tiny effects that result from, from like this small kind of marginal change in your, you know, in your overall marketing mix. And I think they missed the forest for the trees. So I would say prioritize answering fewer, bigger questions less often and versus answering everything. You know, if you can get those big questions right and you can have a framework to answering them and re answering them over time, then all of the little things that you do, maybe more intuitively or using less rigorous methodologies, you will see in totality are they impacting those macro KPI's that really, really matter. And the other piece of it is patience. Right? Marketers are now so obsessed with I want to run a test today, yesterday. I want to see the results and I want to see a 2X. You know what I mean? Like it doesn't, it doesn't work that way. Marketing in most cases is a big tanker that takes time to turn and you know, you can only know over sort of longer periods of times if the, the things that you're doing on a shorter period are impacting those, those trends and so understanding, you know, kind of zooming back out, being patient, understanding that the day to day optimizations, which by the way, sidebar I don't think should be done as often as most marketers are doing or even week to week ones we can get into that.

But it feels so good to turn the lever.

It's such a visceral thing. Let me tweak this knob. Oh, I just see this creative just got extra reach like. Yeah, but it doesn't work. I mean the research shows that and the way the algorithms work doesn't work. But that's a sidebar in any case, you know, it's the, the aggregate, the portfolio impact of your, of your decisioning that has to have some kind of, you know, Maybe slightly less rigorous just because you're forced to do it that way. And then reassessing that on a, on a macro level, on a, you know, biweekly or monthly basis, is that trend going in the right direction? And if so, let's lead into some things a bit more. Okay, we've overcooked it. Let's pare back on some things. It's a constant push and pull, but the pushes and the poles have a delayed impact and have a delayed, you know, ability to kind of for you to be able to assess how well those things are performed.

I think it's a really great and thorough answer. And um, I really like what you say of like really focus on the high leverage things first. And I think that that truly is the most important. The other thing that I, the advice that I end up giving a lot that often surprises people is like, I'm a data person. I've been doing econometrics, data science my whole career. I often tell people like, there is no test you can run to answer this question. Like there is no data you can get. You're just going to have to make a strategic decision. And I think that that's sort of like, you know, even the next level of data driven is like recognizing that you can't answer every question with data. Like your business cann 100% driven on data. There are decisions that you have to make that are based on strategy or gut instinct or like positioning. Like there's all kinds of different things that just cannot necessarily be answered easily with like a very crisp experiment or a very crisp piece of data that you can look at. And that's okay. You just have to like be savvy enough to recognize that like look, this set of questions we're going to, or this product direction or this marketing direction we're going to go because we think that it's right and because it matches our like long term strategy. And we're just going to have to like make that decision and that's okay. And again, we're going to use data where we can to help us optimize the business, but we can't use it for everything.

You know Michael, I'm laughing a little bit because if anybody goes to my LinkedIn and you read what I have in my about section, I have four lessons that I, you know, kind of like a four bullet thing that I feel are lessons that I've learned. One, scientific thinking is imperative for good decisions. Make hypothesis, get the right data and validate. Number two, in spite of one Sometimes you need to take a calculated risk. Science has its bounds and on occasion, if the pool looks clean enough and deep enough, you need to take a plunge. And so I think we're both kind of saying the exact same thing. And I totally agree. Yes, it's knowing when to use experimentation in science and when it's reached its limit. And you know, you cannot, you cannot, you have to strike a balance, cannot go too far one way or the other.

Totally agree. And now, speaking of things that are difficult to measure, brand advertising, a lot of people on LinkedIn talking a lot about brand. I think a lot of people use that word to mean different things. A lot of debate happening. So brand advertising for you, like, what is it? How do you measure it? Should you measure it? How do you think about that concept and sort of these ideas that people are talking about a lot today?

Yeah, I don't believe in brand advertising. I believe in performance advertising with different time periods and different strategies by which you can elicit somebody to purchase. And then it could be a direct way to purchase or it could be an indirect way for somebody to purchase. So I think it's a, it's a misnomer to the kind of like in an artificial barrier that you're creating. Yes. You know, I think that brand, if I'm to think about classically, what does brand mean and brand advertising, it's a set of strategies that are not driving somebody to necessarily buy your product or your service right away, but are designed to put your company's, your company, top of mind for individuals so that when they do have the opportunity to purchase, they're thinking about your company and they're going to have a tendency to buy your product. So I think largely modern mmms, and you can probably speak to this in even more detail than I can capture that impact over long periods of time. If you're designing the models correctly. I think you can also test into whether or not such strategies work by having holdouts and longer term holdouts and things like that. So it's very, very testable. But really I think it's more of a mindset. What type of strategies are conducive to your business? Is it more the in your face, here's a promotion, you know, buy it right now, which can work. It's not a, I'm not disparaging that at all. I'm just being a little bit hyperbolic about it. But those work and a lot of businesses have been built on that. Or is it more of a, here's who we are here's what we do, here's a, here's a, you know, a story about our company and does that over time yield better results? And I think that, I think it's different for each business trends that we're seeing. And I'd be curious to see if you know from the work that you're doing, you're kind of echoing this is we are seeing that there is generally a tendency for companies to think to move away from that really hardcore in your face doctor Advertising and pare back a little bit and lean into more of the, you know, the upper funnel type strategies and approaches. And in fact the work that we've done, the models that we've built and the tests that we've run are supporting that can be a very effective strategy in a lot of cases. My hypothesis as to why this is, I think the, the digital algorithms that delivery algorithms are getting too good for themselves and they're really, really good at advertising on and optimizing on last click and on, you know, basically non incremental metrics. And that's finally starting to manifest itself in, in the models and the test that advertisers run because more advertisers are running those tests now relative to a year or two ago. We're starting to see that trend emerge more. But I'd be curious to get your take on that. What, you know, if you guys are seeing the same thing or you know what's.

We definitely see a mix. We work with a lot of advertisers that run, you know, full funnel marketing and so they do sort of very bottom of funnel, very sort of direct response oriented marketing, but then also do a lot of TV and video streaming, tv, radio, other types of other audio formats. And I think when I'm talking to people who are asking the question about brand, what I often say is, I say like look, I don't think similar to you, like I don't think it's a good idea to be like we're going to carve out a section of our advertising budget, we're going to call it brand and we're just not going to measure it. I'm like that's generally a bad idea. I think that's a recipe for a lot of wasted dollars. But I do believe that there is often there can be not even often there can be these very long term effects of certain advertising that are very difficult to measure. And I think it's okay again to accept that there's going to be some of that that's very difficult to Measure and I think it's reasonable to build into your marketing program some investment that is building up that you know, again, long term brand mental availability, whatever you want to call it, but you do want to be thoughtful about exactly how are we going to measure it and how is this fitting into like our overall strategy of marketing. And so generally what I advise is I tell people like look, if you have good reason to believe and good sort of strategic reasons to invest into this, like do it spend money on tv, maybe don't hold your TV advertising to the same sort of short term or medium term performance metrics that you do to your other channels. So maybe you just have a different cost per acquisition target or a different incremental ROI target for TV than you do for Meta or for search and then make the bet and continue to invest. And I think that that's like a totally reasonable thing to do and say like look, we think that you know, we're going to run TV to break even and we wouldn't do that on our other channels. But that's, we're going to make that investment because we think there's going to be some long term impact that we're not going to be able to measure effectively. And I think that's a totally reasonable approach. But be thoughtful about it. Don't just be like oh yeah, the agency said that this was going to be a big brand campaign and we're going to spend a ton of money and we're not going to look at any metrics that I think is a real recipe for disaster. Instead you want to think about everything should be performance marketing. In to some extent it should be driving purchases. If you're not driving short term purchases, you're probably also not driving long term purchases. There's been a lot of studies that have shown that. But you can be thoughtful about. Look, exactly what is the like target metric we're going to, we're going to drive this towards and how do we want to, we don't have to think about every single marketing channel has to have the same ROI like short term ROI as search does. That probably is also not a good recipe for building like a very like large brand over a long period of time.

Yeah, I agree with that largely and most of it. I think that you know, we would still consider and recommend looking at the portfolio approach.

Right.

Like if your portfolio is not yielding the ROI that you need, then you do need to rethink it.

Right.

And you can calculate the time horizon differently. You know, as you include some of those upper funnel channels to sort of directionally take into account, you know, the, the longer term impact that they may have.

Right.

And, and you know, obviously there's interaction effects. Some of those upper funnel channels can cause the lower funnel channels to work a little bit better. And so, so really all of that needs to be taken into account. And to your point, sometimes you cannot directly measure that, but you know where you can and where you can get directional inputs. We're big fans of, of Heidi Howe for example, looking at longer trends in, in that for those who aren't familiar, Heidi, how, how did you hear about us Surveys usually post purchase surveys or sometimes not post purchase. Sometimes they're kind of like midway through the funnel. But, but those could be really interesting data points. Even including those in your econometric modeling sometimes can be very useful way to directionally get at some of the brand impact that it's having. You know, brand lift studies I'd say are interesting but, but oftentimes I think folks over focus on I need to drive a 2 percentage point increase in brand lift. Okay, what is that doing for your sales? And, and you know, if you think about some of the large CPGs, they have done the math to understand what is the causal impact of a percentage point delta in some of those brand metrics. They understand how that impacts sales.

Right.

Most advertisers who I have seen working with brand studies have not done that work. And so you're often chasing what I would argue is somewhat of a vanity metric.

Right.

In those brand lift studies. And again not to say they're useless, but you have to know how to use them and you have to know how do you, how do you do the subsequent analysis to ensure that if you are driving changes and brand metrics, it's having a causal influence on, on your, on your sales. And at the end of the day measurement is, it's a, it's a cocktail.

Right?

There's, there's no, there's a, plenty of great tools. Obviously you know, recast one of those great tools, couple of others, you know, your competitors, I think all have their, their, their benefits as, as measurement SaaS. But my opinion would be, and the reason we started a measurement consultancy versus you know, building a product directly, although we do have some proprietary tech that we employ, is that it's an ensemble approach and you can find great tools that solve certain pieces of it. But at the end of the day you're going to need to do more than just look at a dashboard that shows up. You have to think critically about those numbers and supplement them and sometimes, you know, put pieces together and build that story out of a lot of different data points and a lot of different analytical approaches to really get the full picture of your marketing impact.

Yeah, I think that's exactly right. And I think again, just to summarize a little bit of what you were just saying is things like brand lift can be important metrics to look at, but they're not an end in themselves. Right? And so you always have to be thinking about whatever metric we're looking at, how do we tie that back to some idea of incrementality, right? Of, hey, when we make this change, this is what eventually it's going to, how eventually it's going to impact our, our bottom line, because that's the thing that matters. And I think a lot of organizations end up getting distracted by these alternative KPIs. Like, yes, we drove a ton of brand lift, but then no one did the extra work to think about. Okay, well, like, did that, Is that actually going to drive profit for us or is that just going to like, go away over the next three months and isn't going to matter because people like, you know, aren't able to purchase our product or we're not competitive on price or whatever. The thing is, the things that actually matter when it comes to saying, was this a good investment for us or not? So we've talked a bunch about media mix modeling, incrementality. You know, whether that's experimentation, there's, there's different ways to think about doing it. How do you think about, like, when are brands ready for these tools? You know, I get a lot of brands reaching out to me and, you know, they're often like very young, just starting out. And I often say, like, you know, you're not ready yet. Like, you should be focusing on other things. I'd love to hear like, you know, again, as a, as a consultant of this space, where, how do you give people advice on, like, what's the measurement journey that they should be going on? When should they be thinking about, you know, investing in these different tools, especially for like, startups, you know, D2C startups that are, you know, have started in the last one or two years, where, when, when should they be thinking about investing in these tools and building up like an incrementality practice?

Yeah, I, you know, I would say there's a couple of points here. One, you're never too early to start thinking about incrementality, right? You don't need to have massive tools and really get crazy about, you know, formally testing things. But I do think that, that from day zero you should go into the mindset that as my business scales, as it grows, it's going to be harder and harder to use out of the box, you know, platform metrics and things like that to understand what, what, what my marketing is doing for me.

Right.

And, and I think that incrementality, whether it's really coarsely estimated in the early stages and then later on, you know, maybe it's got more precision around it. You should be thinking about incrementality like I said from day zero. So if you, if you got one channel, let's say like a lot of D2C startups or Instagram, kind of going all in on Instagram and, and finding great growth with that. The reality of it is probably most of your marketing in the early stages is going to be incremental.

Right.

Because it's the only thing you're doing and you don't have any brand recognition. So by, you know, where else could the customers be coming from? There's like it's going to be, it's going to be incremental. So, so you probably don't need to do anything every now and then. If you want to do something like I'm going to shut it off for a week and see what happens and then really test it out, fine, go for it. But I don't think you necessarily need to overburden yourself when you start expanding into more than one channel. That's where I think it gets interesting and where you want to understand the interplay between those channels. I think honestly the sweetest spot for a DTC startup would be not a ton of brand recognition yet spending. Call it around you, you know, five to $10 million across two to three channels total, every, everything approximately the same media weight. Like you don't have 90% in one and you know, five and five in the other two.

Right.

But, but that like kind of sweet spot that I described means you can be really creative with how you test. You can do, you know, different type of multivariate testing in a very sort of low weight environment. You can really understand the interaction of the channels and lean into that. Where, where I think, you know, the next phase is when you start, you got multiple channels, you've been around for a few years, you're doing some upper funnel work, explicitly upper funnel work. That's where I think you need to start exploring an econometric model and coupling that with incrementality testing. I think incrementality testing needs to be the core of what you do, but it becomes insufficient as the sole tool when you've got these multiple channels and when you need to make more month to month decisions. You know, a lot of great advances have happened in the last few years in MMM and econometric modeling and that allow you to, they're not the old school, they're not your granddaddy's mmm, right. These are sophisticated Bayesian models that you can update with experimental results. And you know, and so these are, these are, I think should be in every advertiser's toolkit once they've reached that stage where they're multi channel and they've got more complex questions to answer. But loosely, I would say that's the progression. If you're an early stage startup, again, I wouldn't, I wouldn't overthink it, but just, just make sure that, that you, the only thing I, I think you do need to think about is incrementality. Have that baked into the core of what you're doing. And again, that doesn't mean you need to be explicitly testing everything, but understand what it is, why it's important and when, you know, when you need to start to worry about it a little bit more.

It's a great answer. And it feels like it's sort of exactly the same thing that I've told so many brands on various different phone calls over the years. Like we are 100% aligned and especially the first part so many people skip over. But I think it's, it's actually the most important, which is like building the idea of incrementality into your culture. And you're totally right. Like that doesn't mean that when you're spending $10,000 a month on advertising, you need to be running all of these different incrementality tests. That's probably not like the best use of your or focus. But as you are thinking about marketing effectiveness, you should always be asking the question how much of this is incremental? And at the very beginning, as you noted, it's probably a lot. As you start to scale, you want to keep that question front of mind just so that it's built into your culture. All your marketing managers know that when they come to you with a report on performance, you're going to ask them how much of this is incremental. And once that's built into the culture, it helps you avoid so many problems. I talk to these brands, you know, fairly regularly. It's like they're spending $40 million a year on marketing. And the CFO is obsessed with their Last Touch attribution report and now they're just like, they're, it's like so painful to make a change because the CFO has like gotten so obsessed with this report that like now they want to do anything that isn't, that doesn't drive like amazing Last Touch roas and they can't invest in it. And so if they would have like from the very beginning had you know, better healthier conversations about what incrementality actually means, they wouldn't be in this situation. But now it's so painful they have to go to the board and explain this stuff and it's like at their scale it's really hard to turn the ship because they never had these conversations like earlier in the life of the business.

Totally. And by the way Michael, maybe we can use this as a quick psa. So marketers, probably mostly marketers that, that are your listeners, I want you to send this next piece to your cfo.

Okay.

Last, last click is wrong and it is a waste of money. I don't care how good it is and how easy it is to throw into your forecast models, it is wrong. And we need to get CFOs comfortable with dealing with you know, error bounds and statistical methodologies. You know we at growth by Science, I'm sure at Recast as well, we're all happy to talk to CFOs and explain it and help get folks around that. But I do think that, and maybe you guys feel the same way. A lot of times the non marketing org folks are where we get, you know, hamstrung in the process of evolving, you know, clients measurement practices. And so you know, let's, let's, let's try to get everybody comfortable with the idea of experimentation of how to work within, you know, the stochastic realm. And, and it's, it's so while it is challenging, I'm not saying it's not, you know, it's the easiest thing in the world. The trade off is much better than using the wrong, the wrong data.

Right.

Last click needs to be thrown away. There's really like unless you're an early stage startup marketing on one channel, I think it's virtually useless.

And it's so I am not going to go that far. I don't, I think that like some sort of touch based attribution is very reasonable especially for the marketing team. Like I don't think that you should, the marketing team, I would say like probably don't share that out like with the CFO for the reason that you're saying, but like within a marketing team. Looking at your touch based attribution and seeing trends I think is a reasonable input into your decision making process, but it's not a measure of incrementality.

Yes. So I agree that the touch based attribution, kind of what we were talking about before, using something suboptimal to guide you on, on the more like micro level optimizations, I agree, can be more useful than that. I think the problem is it's very hard for anybody, a marketer or CFO or whatever to, to, to, you know, separate those two concepts. I'm using this directionally. I'm not looking at the absolute value of my last click conversions for this particular, you know, campaign or whatever, but I'm, I'm going to see trends and make decisions on how to distribute budgets between campaign A and B based on that. But I care more about my performance at a macro level that is not measured any way to do with Last Touch. I think that the reason why I would say it's very dangerous is that those two kind of ideas end up colliding and you get a lot of bleed over both ways. You get bleed over of like, okay, well I'm just gonna look at the total channel last click attribution and I'm gonna make big tens of million dollar decision based on that or the other way which is not as common but we kind of alluded to earlier in discussion as well. Let me micro analyze and create tests on micro tests on like these small things that are really, really hard to discern using a scientific experiment. So you know, I think that's, that's where I would say we are in alignment where yes, using that. But, but the big caveat here is don't let that bleed over. Be cognizant that it's two separate things.

I think it's a really important caveat and I sort of, I see your point of being like oh man, it's almost better to just throw it away because the bleed over is so common. And like again, I think I do believe that like digital tracking approaches, whether it's mta, first touch, last touch, I think they can, they should, they can be important inputs into decision making. Like I think it's worthwhile to look at those reports, but only if you can actually separate that from incrementality and make sure that you're keeping in mind that this is not incrementality and it can mislead you if you don't use it correctly. But I agree that I've seen enough organizations not be able to make that separation that it really, it puts a.

I haven't seen one. And I will say this as a counterpoint to what we're talking about. I worked with one organization where we, we explicitly threw away all things. Last click. We didn't do anything. Last click. We didn't put any pixels on. We didn't do any.

Happy wow.

Full thing.

You did the full thing.

Oh yeah, we did the full thing and it worked great. We did, we did pure incrementality testing only we didn't even have an econometric mod.

That's amazing.

I've actually, I've done a couple of talks at a few events where I've sort of advocated for this like extreme situation. Not to say that everybody should do it or that in all cases it's the right thing to do, but it is possible. Certainly possible. And I would say, you know, consider reducing the reliance on these post exposure attribution in all things. And I'll give you some justification for this based on, you know, just, I would say a lot of knowledge on how ad tech platforms optimize and folks can go to my LinkedIn and see my background and you know, take a gander at where this knowledge comes from but you know, there. And maybe it's changing over time. So we'll caveat that but historically platforms have become excellent at predicting your behavior. Okay, they're great at predicting your behavior. They're also optimizing showing ads based on post exposure attribution.

Right.

So I've seen an ad and then I converted and I'm going to show, you know, I'm going to give that as the feedback loop. So when you couple the excellent capability to predict behavior, the post exposure optimization and then the post exposure attribution, you get a self fulfilling prophecy that is great at optimizing revenue for the ad platforms and not necessarily optimizing outcomes for the advertiser.

Right.

And so that's why I've got like a anything last click. Scares the heck out of me if I'm very honest. And you know, rather, I'd rather, you know, honestly I'd rather rather folks optimize on Heidi Howe results than last click. So even, even that I think would be better.

So for what it's worth the recommendation that I give to people is I say the report that you should build is you should, if you want to do like a digital tracking type report is you should have a report that updates every week and it shows, you know, either implied roas or implied cost per acquisition. First touch attribution, last touch attribution and then Heidi, how so like those three side by side refresh it every week, think about like what it implies. That first touch is so different than last touch for certain channels and Heidi Howe is different from those. And then again just use that as an input. And I think that, I think is a, is a decent recommendation because it says like, look, there's three different numbers. They're all going to be changing constantly. You shouldn't fixate on any one of them. None of them are correct, but they all give you some information. And so I think that that recommendation I think strikes a little bit of the nice balance of not being like none of these are correct. None of these are our source of truth for reporting. They're all interesting, they all give some amount of information and we need to use all of those data points to actually make, make the decision and think about how we're going to steer marketing.

I like, I like that approach in general. My only concern with that is just the quality of data that goes into the first and last touch. And you know, again, this whole area has gotten muddy, less muddy, more muddy. But the tldr, you know, this, this, this like holy grail that marketers have had in the past that I want to understand what everybody has, what interactions everybody has had it give me some tiny timestamps, precise timestamp when they've had it and I want it comprehensive. And if I had that data set, oh man, I could just tell you the incremental, I could tell you what, what the value of all of my marketing is at the super granular level. I think, I think it's, it's a, you know, just, it's just like an impossible thing to, to arrive at. First of all, that data set cannot be built. It just simply cannot be built. There's, there's huge gaps always in that cross device attribution and you know, with lack of cookies on safari and things like that, just you cannot build that, that data set. And even if you could build that data set, I would, I would hazard that most marketers would use the wrong methodologies to try to arrive at, you know, business logic type stuff versus actual, you know, causal methods. But anyway, that, that sort of, I've got these, I've got this like visceral reaction as you can tell, to, to anything that's post exposure. And so I would say very cautiously would be my advice. I think what you're suggesting, Michael, is, is probably this as far as I'd be willing to stretch, you know, but, but yeah, in general I would favor less, less of that and, and also like less investment. I mean this is the other piece of it, right? The place where, and another reason why Post Exposure really kind of gets under my skin is I've seen the level of investment that a lot of marketers do have made in trying to amass that holy grail data set. And it's crazy high. I mean huge teams devoted to getting cleaning data, connecting it, multiple ad tech constituents, piecing this like Frankenstein of different platforms and services together to get this data that ends up still being really wrong and frustrating. And I would say like do 1/10 of that and take, take the rest of that, that time and money and energy and invest it into, it's, you know, into into you know, scientific approaches. I think that has a much higher roi.

I totally agree with that. And this conversation is sort of teeing up the last topic that I want to hit on. You've worked at a lot of the, the ad platforms in marketing analytics, marketing measurement. And I think you know, our listeners want to know like are these marketing analytics partners at the big platform, are they just trying to sell more ads? And maybe, maybe not. But is there a good way that brands should be working with these partners? Like how can you like if my meta rep is calling me constantly or you know, constantly saying we should talk with our, their marketing science partners, like how should brands get value out of those people or is it just like don't take the call?

I mean like this is a, this is a really interesting question and I say this, having led you know, team teams like this and help build these teams, I think there are a lot of smart people on, on the marketing science side. Depending on how they're organized, some of them are wrapped into sales teams, which means that they're inherently, they're more explicitly, you know, comped and rewarded for increasing spending. Some are not, but at the end of the day all of them have some kind of stock based compensation and are, have a vested interest in the platform being successful.

Right.

Those teams tend to attract non salesy people. So I do think that, that folks on those teams are genuinely trying to help advertisers be better. But even with the most altruistic person in that, you know, in that role, they're still limited by only the data that they see for their platform.

Right.

You're typically not sharing other platforms data with them. And, and I, I think you'd be hard pressed to find any account executive that would say, take a result that a marketing science person said hey, I think they should be spending 20 less in a platform. That account executive will say, great, really cool. You stay over here, do not come on the next call. And so, you know, like you, you're. And maybe it happens from time to time. I have never seen it where you recommend somebody spend less on your own platform. And so all this to say I think there's, there's value in speaking to, to smart marketing science folks at all these platforms. You know, having worked on these teams and built these teams, I know that they're generally high quality folks that are there but you know, just understand the motivations, right? Like the same way I wouldn't necessarily recommend, and I don't recommend looking at platform dashboards and as a source of truth for everything, do your own stuff. You know, do the same thing with, with, with the marketing science folks. Take the input, take it with a, with a grain of salt. And I think it's extremely important for advertisers to take responsibility for measuring their own, you know, marketing impact.

Right?

Don't, don't just rely on your meta or Google or Snap or whoever rep to do it for you because while it might be right, it's, it's still going to be biased just by the fact that it, even if it's unintentionally biased, it'll still be biased. So don't definitely don't not take their calls. I mean I think they, unless they're annoying then, then you know, don't take any call. But I think they're, I think they're smart folks in general. And again, it's just take it with a grain of salt. I will say this. I am not necessarily a huge fan of exclusively relying on platform incrementality tests. So I know a lot of platforms do have them and while I think the science behind most of them is pretty decent, there's a lot of reasons which I think we would need another hour or so to go into. But the TLDR is you definitely use them. But same as when you're taking advice from the marketing science folks. Take it with a grain of salt and better to have something that's more apples to apples comparison. A geotest. Typically you can do really, really well with the geotest if you design it correctly. There's all kinds of open source tooling out there and paid tooling that can help you design and execute a really, really high quality geotest that I think will probably be more valuable to most advertisers than exclusively relying on the platform incrementality test.

So is the recommendation like, look, this sort of user level platform increment incrementality tests can be good, but worth double checking it with external tests at least every once in a while. Is that sort of how you frame it?

I would actually do it the other way around. I would, I would make geo tests be the core of your, of your practice. And knowing that there's some limitations that if you're not spending, you know, if your budget on platform x is like 5, you know, 1% of your total budget, it's going to be harder to get a read on that and so be okay with some of those limitations. But I think, I think geotesting is and has, you know, there's been a ton of work that's been done in this last few years. We've used it to great success over tons of clients. Again, just know the limitations of where it can work and where it can't work and what questions it can answer and what questions it cannot answer. I would only use a platform test, you know, in particular instances and, and if you feel like there's a specific nuance that you want to, you want to understand and I would actually rely on it more for like a relative comparison of tactics within a platform than the objective lift of a platform over and above the rest of your marketing spend.

Andrew, this has been such an amazing conversation. Thank you so much for joining me and thank you for sharing your wisdom with our audience.

Thank you for having me. A great conversation. Echo that as well, Michael. And thanks for having me on and excited, excited to keep, keep fighting this good fight, you know, along with the rest of us here in measurement land.

We will, we will surely keep fighting there.