Avoid A/B testing pitfalls with Eppo CEO Chetan Sharma | Coffee Breaks

Foreign.

Hello and welcome back to Recast Marketing Measurement Coffee Break series where we dive deep into complex marketing and measurement topics with industry experts. This week we're talking with Chetan Sharma, CEO of epo, an AB experimentation platform for modern data and product teams. Che covers a lot in our talk. What companies get wrong about experimentation, how to build trust in your test results, the subtleties of P values, how to manage uncertainty, high leverage impact testing, contextual bandit algorithms, and a whole lot more. This is a really value packed episode that data driven marketers, data science folks and executives will get a ton of value from. So let's jump in.

All right.

Welcome back to the Marketing Measurement Coffee break with Recast. I am very excited today to have have with me longtime friend Che Sharma. Che is the CEO and is it founder or co founder?

Founder, Solo founder? Founder.

Founder of Eppo. It's just an amazing company. They do amazing work in experimentation. I think Che and I see a lot of the world the same way and how we think about experimentation and the value of it. So I am really, really excited to have this conversation. Jay, welcome.

Yeah, it's good to be here, Michael. Yeah, it's fun. We first met each other right as I was getting FO off the ground. So it's crazy to see how much things changed.

It feels both very different and also very much the same. So I'm excited to be continuing on this journey. As always, we try to keep these sessions very information dense. So we are going to jump right in. So to start, actually, given that our audience is mostly like marketing analyst types, they may be a little bit less familiar with EPO and what you'll do. So can you just give like the quick two or three sentence background, what is epo? What do you do? Where do you, how do you help companies?

Yeah, definitely. So we're an experimentation platform. You know, a lot of it comes out of my experience in data science coming from Airbnb. And the thing we really focus on is experimentation as a broad mindset, as a cross cutting workflow for a whole organization to build, you know, to actually deliver the velocity growth that experimentation is supposed to do. So, you know, the thing that's kind of our calling card that's a bit different is, you know, we try to do a lot of types of experiments, not just a marketing website test, not just a product launch or performance marketing performance test. We try to kind of do it all. And the way we can do it is because of our unique relationship to data and warehouses, stuff like that. So a lot more to get into. But, you know, I'd say experimentation is a broad mindset. You know, how do you build winning companies that actually get velocity and growth from it? That's what we focus on.

I love it. First question, what do most companies get wrong when it comes to experimentation?

Yeah. I think the thing I see the most is people just vastly underestimate the opportunity cost of getting it wrong. And there's kind of two elements to that. So in terms of what are people getting wrong for experimentation to deliver velocity and growth? The whole point you're doing is velocity and growth. For that to happen, you need to be experimenting pervasively. It needs to be across a lot of different use cases. If you're only experimenting on a marketing website, then you're not really experimenting. It has to be fast. If it's not fast, then people just won't experiment. You won't get it pervasive, and you also won't be able to test many ideas. And then it has to be trustworthy. I think that last part is you got to avoid the pseudoscience, avoid statistical theater. I mean, number of people who live in a point of cognitive dissonance where they both know it's wrong and that no one trusts it, and yet they still kind of do the motions is astonishing. The, you know, you're advocating for this.

Sort of more holistic approach to experimentation. Can you talk about, like, what are those other things that a company today that, like, is really focused on testing the website maybe should be considering doing?

Yeah, definitely. So I'll talk about some broad buckets and then maybe a few kind of iconic examples that'll kind of open. Open the aperture here. So in terms of broad buckets, I, you know, again, the marketing websites, one another is just product changes, I think, you know, one of the studies I always like to call out is there have been a lot of studies coming out of Google, Facebook, Airbnb, Amazon, that basically show that of all the product changes that are made, only, like 20% of them are actually improving metrics in a meaningful way. And that's a very standard industry rate across all those companies, certainly among most of our customers. You see that when you actually take the time to measure things, something like 20% or up to a third basically are going to actually move metric positively with another third probably degrading the system without you even knowing it. You know, it turns out when you look at the sheer diversity of your user base and all the segments that exist within it, you're probably degrading it for some people without Knowing it. So, you know, that's just a lot of wasted time, A lot of wasted time, you know, building things that don't work. So that's on the product side, on the marketing side. And you and I both exist in this world of, you know, wasteful marketing spend. That's another place where you kind of need to know what's working, what's not. And as listeners of this podcast know, that's a big, deep conversation. So it's important to get the science right. You know, we live in a world of AI. You can't do a podcast episode in 2024 without talking about AI. That's another place where, you know, these gen AI models, they're, they're incredible in how much power they have, but they're inscrutable in how they're working. New York Times ran an article that said like the hardest problem in AI is measurements. And the reason is that all these models, they spit out like raw language. At least the old machine learning world spat out numbers and you can know whether a number is correct or not is too close to four sort of thing. In the world where you just get a sentence, you don't really know it's good sentence or not. And so it just leads to situations where out of thinking or making the right decision, you just go for the most expensive model, the thing that has the brand name, and not actually have any idea of whether it's improving things for your customers. So there's wasteful spend on the order of tens of millions of just paying up for the latest sort of model, as well as the lack of customer intelligence where it's like, I bet that some core group likes this AI model and I bet there's some type of language speaker, type of device context that does not. And ideally you could better do that. So it costs a lot. If you just kind of stare at most of your important decisions, there's usually opportunities to de risk them with an experimental approach.

And in sort of your first response to my first question, you mentioned correctly, statistical theater, I think was the phrase that you used. I've definitely seen this and I think there's variations of them. Like I've seen the case where they're doing what there actually is, trust, but the statistics are done poorly. And then on the other side that you also mentioned where it's like there's actually no trust and people aren't really sure if it's working and there's not buying any failures, your modes that you've seen here.

Totally. I mean, look, this is Probably a whole podcast episode in itself, but I'll touch on a few of them. I mean, in terms of where do you need to get trust again? The goal is it needs to be pervasive. It needs to be fast and trustworthy. If it's going to be pervasive, then leaders have to believe that all these reported metric lifts are real. And I think if you polled the vast majority of CPOs out there, they would not believe it. This team said they boosted revenue by 10, 15% in aggregate, and I just don't believe it. And the funny thing is, they're probably right based on a bunch of different reasons. So at the aggregate impact level, the whole point is to say you drove velocity and growth, and people usually don't believe those results. At the experiment level, what is the quality of your scientific process? There's a bunch of issues here, the sort of things that come up first. There's a bunch of the integrity of the test. There are problems like you think traffic imbalances, you think it's 5050 test. It's like 49, 51. And it turns out that's deeply problematic. There's problems of using the right metric. A lot of people think experiments are supposed to just drive kind of clicks or something like that. And that's usually just an artifact out of how vendors used to be architected in the past. If you go and look at an Airmeer or Netflix, whatever people are, the metrics they use are the bottom line. Bookings for Airbnb subscriptions, for Netflix, time on site, that sort of thing. Stuff that is like, if you move the metric, the outlier is swinging the entire test. And outliers are just, you know, usually weird blips that are indicative of anything else. You know, again, the whole podcast was that we could probably do. But you know, what ends up, what's interesting that ends up happening is either you exist in a state where you, like, don't run experiments because no one trusts it, or you exist in a state where you just kind of hire your way out. You know, you just start trying to stack up experiment specialists and you make them a bottleneck on everyone else. So, you know, one way or another, unless you get the science right, the process right, you don't end up getting it right, which is pervasive, which is fast and trustworthy, and then that ultimately means that some other company is going to unlock that trio and probably out compete you.

We're running lots of tests and you expect that sometimes you will get statistically significant results that aren't actually real.

Right.

If you're using a P value of.05, you would expect that, you know, one out of every 20 or 5% of the time you're going to get false positives. Does that sound right to you? Are the things that we were doing wrong, is that like the right way of thinking about these tests?

Yeah, those are both valid reasons. So, you know, what you're alluding to is like when you're saying could be a blip, there's, there's problems of generalizability. You know, when you run an experiment, it's over a short time period under certain circumstances. And you know, in the ideal, certain science world, you replicate. Right. That's how, you know, we build up knowledge in medicine, in other fields. And then their other problem is like, you know, they're. If you're running so many tests, I'm going to show up as positive. The thing I'll just say is that if you want to get the process to be trustworthy, you have to solve it at the experiment level. I would say most companies, the fundamental integrity of their tests are just wrong because there's just enough stuff to get right that usually for most companies, they're probably messing up something. But the other problem is if you want to get aggregate impact, if you're trying to figure out how to generalize it, you need special tools for that. Like it's actually a problem distinct on its own. Right. You know, to give you a sense, like what we do here at EPO is, you know, first you have to extrapolate it to like a global lift. For example, if your Test is a 10% lift, but it's only on 5% of your population, then like, you know, it's not a 10% lift, it's substantially lower, you know, 0.05% sort of thing. If you. And then what's more like there's this, you know, one of the funny artifacts of frequentist testing, this getting lope into the weeds for the typical methods people use, is that it never promised. You can just add up point estimates like that. That was never actually part of the math. It gives you very rigorous judgments on whether to launch something or not. But it's structurally biased in terms of the point estimates it delivers. And so you need again, shrinkage corrections. We actually are big proponents of Bayesian aggregate estimates which have natural shrinkage built in. Or you just kind of run an experiment on the program in general, things called holdout groups, which are meant to say, instead of trying to sum up experiments in any Sort of way, just run a sort of meta experiment where 20% of the population is going to get no experiments for a quarter and you just compare that against all the winners. So again, you need special tools if you want to get that sort of leadership buy in trustworthy aggregate impact. And I think just a lot of people, you know, in terms of what does it mean to get it right? It has to be pervasive, it has to be trustworthy and it has to be fast. You're not going to get there without also treating that separate problem of like, what is the accurate impact, you know, with a certain amount of seriousness.

Yeah, I love. I think this is one of those things where it's like the AB test seems really simple, you know, And I think a lot of like product managers go through the boot camp or whatever and they're like, yeah, it's an A B test. Okay, great. Like I know how to interpret the P value and determine if it's statistically significant or not. But there's a lot more complexity under the hood and a lot more like subtleties in ways these things can go wrong that you just need to have someone thinking carefully about whether that's someone internally or a vendor or whatever. Like, there's just a lot of complexity here that lies below the surface that is, you know, it can totally lead you astray if you aren't getting it right.

Totally. Look, the. It's actually one of the blessings is that the cool thing about a B test is that it's actually got this like fifth grader level. Like, I generally know what we're doing here. You know, divide people in two groups, see which one outperforms so the other, like, it's got this like nice clean interpretability to it. You know, you've probably been running something called similar to an A B test all the way back in elementary school. It's just that once you're doing it at scale in like, you know, real data, social data situations, it turns out there's just a lot of things to get right. And look, there's a reason why if you go to an Airbnb right now that the team of people building experimentation infrastructure is like 50 people. It's filled with PhDs and stuff. Like, there's just a lot of expertise if you're trying to get the sort of winning experimentation culture that you see at those sort of companies.

I want to change directions a little bit. So I'm going to ask this question in a vague way and there's sort of a couple of questions Underlying it is more experimentation, always a good thing.

Yeah, you know, it's funny, I, I feel like I've come around full circle to this. Where I first started my career I was like yes, experiment everywhere. And then it got in the muck. I joined some startups that don't have large data volumes and you know, various situations and I'm like ah, maybe, maybe not. And now that I've kind of seen the number of companies out there, I'm actually back to like by. I know there's room to give a more nuanced answer here, but I'm actually want to go maximalist and say yes, by and large we should run more experiments all the time. And the reason is that I've seen some pretty interesting studies that say even if they're somewhat underpowered, if you just deploy more ideas and make decisions in a more Bayesian way, then you're still going to net out to a better performance as a company. Basically in general, the more ideas you can deploy, the better kind of under the vast majority of circumstances. Now because the actual answer is probably nuanced, yes, if you're like a 10 customer startup, you should probably not run experiments and you know, there's room for that. But I'll just say like, I actually for the purpose of discussion think the maximalist position is the more interesting one. Like I think in general, like if you can run more experiments you should. Now I think the reason why people might push back on that is actually the reason I fully agree with which is that you know, experiments take mind share and when you talk about experiments under certain sample sizes or certain types of things like, or certain types of areas where you're experimenting like the leverage is not as high. Right. Like it just doesn't get you as much if you're experimenting on like the simple should I put the panel on the right or left sort of decision then like it's just not, you know, why spend your mind share on it. But this is where I'd retort that good experimentation vendors and I know we're going to talk about bandits stuff soon, but I think good situations will provide a solution that is highly automated where the mindshare is near zero. And you should, for anything that is considered that level of optimization, you should still do a thing called an experiment. You should just not spend mental cycles on it.

So you did a really good job of answering this. I intentionally phrased the question in a way so that I could argue with you no matter what you answered. So I think broadly I agree that More experimentation is better. I think the quibbles that I have are ones that you already laid out. And so I'll say what the two quibbles are. One is just experimenting on using experimentation on small things as a substitute for thinking about the big product things. And so this is this idea of, like, oh, we're going to tweak the background color on this one image in this one page, and like, we're going to run thousands of experiments that are all these, like, little tiny tweaks of, like, we're going to change the font size slightly or, like, move the button slightly. And like, that stuff is. Can have impacts, but, like, it's not going to move you forward as a business generally, like, you need to be experimenting with larger things. And so I have seen the failure case where some companies say, we're going to experiment with everything, but, like, they actually are doing the wrong things. They're experimenting. They're like, distracting themselves and distracting their product managers and distracting their product team by like, doing all of these little small tweaks that are easy to experiment with but not doing the hard work of thinking about, like, are we actually solving the right problems for our customers? Do we need to be taking bigger swings that are maybe harder to experiment with? But, you know, often, sometimes, like, don't feel as good, like, because the logging into the platform and seeing the statistically significant thing feels really good, but is often not driving the business forward as.

Much as I can, 100%. And, you know, I talk to the team all the time about this that, like, you know, when I think of what we need to be in this experimentation space, like, you know, an analogy I like to use is Wealthfront. You know, Wealthfront, betterment, these sort of like, you know, Robo Advisor type things. Have you used those systems, Michael?

I have. I'm actually a betterment user.

Oh, there we go. So for anyone who hasn't, like, basically what these things do is they just, you know, they make a statement to the world that, like, most people build wealth with very diversified strategies of index funds and they just kind of very automatically, like, buy a bunch and rebalance every quarter. Right. And the point is it's highly automated, it's sort of set it, forget it. That generally matches like a very normal investment strategy with idea that you're not really going to get much arbitrage on that. I think of EPO and what experimentation is, is like most experimentation vendors out there, most experimentation practices are operating closer to like a day trader where you're supposed to, like, Babysit in, you know, like individual experiments and overthink every single individual experiment. And so if you think of the day trader like you know, if you are manually, if you're doing experiments like should I put the button on the right or left sort of stuff and you're putting spending mind share on it. It's like this day trader thinks they're getting wealth by like, you know, fidgeting with knobs and stuff every day. It's like that's not really how most people get their wealth. You know, they do other things instead. What you should do is a sort of much more automated strateg of things.

That are maybe harder to experiment with but you know, often sometimes like don't feel as good. Like because the logging into the platform and seeing the statistically significant thing feels really good but is often not driving the business forward as much as 100%.

And you know, I, I talk to the team all the time about this that like, you know, when I think of what we need to be in this experimentation space, like, you know, an analogy I like to use is wealth fronts. You know, wealth from betterment, these sort of like, you know, robo advisor type things. Have you used those systems, Michael? I have.

I'm a, I'm actually a betterment user.

Oh, there we go. So no credit. Like basically what these things do is they just, you know, they make a statement to the world that like most people build wealth with very diversified strategies of index funds and they just kind of very automatically like buy a bunch and rebalance every quarter. Right. And the point is it's highly automated, it's sort of set it, forget it. That generally matches like a very normal investment strategy with the idea that you're not really going to get much arbitrage on that. I think of EPO and what experimentation is, is like most experimentation vendors out there, most experimentation practices are operating closer to a day trader where you're supposed to babysit in individual experiments and overthink every single individual experiment. And so if you think of the day trader, if you are manually, if you're doing experiments, like should I put the button on the right or left sort of stuff and you're spending mind share on it. It's like this day trader thinks they're getting wealth by fidgeting with knobs and stuff every day. It's like that's not really how most people get their wealth. They do other things instead. What you should do is a sort of much more automated strategy of saying look, there is benefit in button left or Right, Red and blue button stuff. Look, there's actual metric list to potentially be had, but they should be completely automated and fast. And what that does instead with all threat stuff does, is say, spend your mind, share on things like, you know, when to buy a house or you know, when to start a company or whatever. You know, there are larger, more strategic, you know, wealth oriented decisions to make. And similarly, analogously with experimentation, you know, we'll talk about like performance marketing spend, we'll talk about kind of larger, you know, questions for the business around like, you know, market, brand power or larger product initiatives and stuff like that. This is where like the experiments are going to be higher scrutiny, they're going to be more collaborative and you need separate tools for that. But that's where mindshare should be going.

Great answer. And this is a great time to transition to a part of the conversation that I am particularly excited about, which is bandit algorithm. So I know that y' all do this at epo and just for the listeners, I'm going to have you explain what bandit algorithms are. But bandit algorithms hold a special place in my heart because my investigations into these bandit algorithms were sort of my first foray into Bayesian thinking and really like this idea, idea of optimal decision making under uncertainty, which is I think a really powerful and different framework from the null hypothesis testing framework. It has really inspired a lot of the work that I've done ever since. So anyways, I'm really excited about the work that y' all are doing at EPO on this. But for the listeners, bandit algorithms, what are they to start? And then we can talk a little bit about how they solve this problem that we were just talking about of like, how do we automate the testing that we need to do?

Totally. Okay, so bandit algorithms, what are they and how they compare to AB experiments. So I think there's two things to talk about. There's first the multi arm bandit, which is the standard case, and then the contextual bandit, which is a kind of more interesting case. So the standard bandit first experiment thing is that under a normal a B test, what you're going to do is you're going to take two versions of the world and you're going to hold them both open, giving equal traffic to both for a month or something, and then you're going to collapse the world into one and that's you're going to declare a winner and that's what you're going to do. What a bandit algorithm is going to do is say, you know, we are going to start like declaring winners early. You know, the second we start to notice outperformance in one group, we're going to start feeding it more and more traffic. It ends up being more automated. And, you know, the big benefit is you can take advantage of winners early. And so there are some situations where this time opportunity cost really matters. So the canonical one is like a holiday promo. If you do an a B test and you don't get the answer to the A B test until December 23, it's not very helpful because once the holidays are done, who cares? Similar deal with news stories. If you are trying to come up with optimal headlines of news stories, once the news stories passe after a week or something, who cares? So you need much faster reaction than a B test will do because your ability to exploit the winning variant decays heavily in time. So whenever you have those sorts of situations, bandits are really, really helpful. Now the special case of bandits that we do here at EPO is called the contextual bandit, which is a version of Bandits that says not only exploit early winners, but also understand that there's differences across segments you have to personalize. So maybe there's not some globally best variation out there, but there are some that are better on mobile than desktop, some are better than with men than women. And when you think of it especially a lot of the types of problems for which I think bandits make the most sense, they are actually ones that have that sort of shape where the difference between groups is actually fairly meaningful and you want to be able to exploit those heterogeneously distributed best. So bandits, what are they doing? They're letting you exploit winners earlier and they're taking advantage of the diversity of your population in use cases to kind of deliver better experiences.

So I love this idea and just to like spell this out for the listeners, like sort of what this implies, which is that if you're, if you're able to sort of get this contextual bandit system working for you, you can effectively get to just like a highly personalized experience for everyone that is engaging with your product. Because the idea is that we know or the system knows, right? No individual human knows very much of this, but the system knows that people who come on an iPhone from this certain geography and have engaged, you know, have this set of UTM codes and have something else about them, right? Engage the most with this page that looks like this with a button over here and this color background and someone else who comes from an Android from this other geography with this other set of UTM Codes is going to have a different set of content with a button in a different place and something else. And the system is able to sort of dynamically generate all of this content for them based on the information that we know about. You know, users who look like this tend to engage more with this set of things. And you move away from this world of being like, we are going to run an experiment and release a feature to all of the users going forward to a world where we don't really think about, like an experiment ever ending. Like, there's just always. The system is just always learning new things and trying out new things and then deploying those new or old things to users that are coming in. So first of all, I'll say, like, first of all, did I get that right, Jay? And then I want to sort of dig into, like, what that implies about, you know, the future of testing.

Definitely, definitely. I think these conversations always get a little interesting because there's the contextual bit and there's the band a bit. Because, like, you could have a contextual AB test where you just run the AB test and you just roll it out according to, you know, what segments outperform each other. You could have a band without the contextualization. As I said, I think that what ends up being cool about the combination is that, like, it's constantly probing, you know, the A B test. Like, basically, if you're going to exist in the world of personalization, you're probably going to hold a bunch of these variations open for a long time. You know, you're not trying to collapse the world from like two into one. You're just going to say, I'm going to exist in the world where like all 10 are always out there unless one just truly sucks for everyone. And then if you are actually existing in that world of like, okay, 10 are going to be out there in perpetuity, then why not constantly explore, exploit, constantly probe? Because it turns out some of these outperformance could be ephemeral, it could be seasonal, it could be something that's going to decay. So the combination ends up being pretty cool.

So I think what's interesting for me to think about is how does the role of the product manager start to change in this world? Because again, like in my back, in previous experience, a lot of the product managers were like, we are going to come up with an idea, we are going to test the idea, and then that idea rolls out and becomes the product or not. But in this world, it's sort of like a different thing. Where it's like the product is actually 10 different things or 100 different things, or thousands of different things depending on whatever intersection of combinations come out. Like, how do you think about the role of the product manager in this sort of new world that we're envisioning, where contextual bandits are sort of doing a lot of this decision making?

Yeah, definitely. And I think there's a few ways to take this. There are some problems to solve to get to that world. I think this sort of contextual bandit centric thing, and it mostly exists in the world where holding 1050, 100 variations open at once causes no debt, causes no issues. So it happens in cms, it happens in AI models when it comes with code paths through products. You'd have to talk to your engineering department of how comfortable they'd feel holding that stuff bug free. But focusing on the places where you can hold many variations open for a while and it's clean and maintainable. I think the big thing that's going to happen is that in terms of product teams, what's going to happen is you're going to focus much more on the policy level than the experiment level. Basically you have a way of making decisions that some weighted function of, of the metrics, right? You care the most about this metric, but don't harm this other one along the way. You know, you're going to kind of demand a certain level of certainty, a certain amount of approval or whatever to roll things out. There's a set of policies you do that kind of like approves the science, approves what goes out. And like, you know, you're going to kind of evaluate like are those the right policies? You know, are those the right metrics? Like, is this actually steering the business in the right way? We want to talk about because like in a sort of contextual bandit world, like what that is basically saying is like you just focus on getting ideas deployed, we'll take care of serving them to the right people. Accordingly, product teams will focus a lot more on idea velocity and then policy level judgment of what ideas are being rewarded or not. So this is actually part of the reason we were so excited about the Contextual Bandits. We think it's our homage to the age of AI where what's going to happen in the short term there's a lot of AI models to evaluate, you know, should I use GPT models or whatever else. But in the long term, you know, what we have seen out of these, these AI models is that they like, they make brainstorming really fast and they make idea implementation really fast. You know, you can create creatives, you can create code really easily. And so, you know, the line I like to say is like, you know, as the. As the cost of ideas goes to zero, the cost of measuring ideas ends up being the bottleneck. And that's where we feel like contextual bandits is kind of the right paradigm to make it fast, pervasive, trustworthy, invest in a world.

I love the vision. For what it's worth, this is like a world that I'm very excited about. I almost started a company that would have been a competitor with yours of, like, building these contextual bandits. I think it's very much the future of how digital products are going to be built, and I think it's really exciting. I love what y' all are doing. Changing topics a little bit. I mean, we've been talking a lot about sort of experimentation in general, but a lot about, like, in the product world. And a lot of our listeners are. Have more of a marketing background. So I guess, like, I want to hear, you know, maybe like, at epo, like, how much are you mostly working with product teams versus, you know, our marketing teams using. Using epo, and then how do you think about, like, what marketers should be doing on the experimentation front?

Yeah, baby, yeah. So, you know, what, we have product people, AI people, marketing people using epo. You know, the largest block today within marketing has been lifecycle marketing campaigns. You know, things like email. Yeah. Emails, notifications, that sort of stuff. That's what we see the most of. You know, there's also some stuff that meshes with product as well. You know, these sort of omnichannel setups. Like, we see some of that as well. I think, you know, how should marketers be experimental? The way I see it is marketing. Marketing channels and campaigns have the same shape as product, where, like, things are pretty uncertain of, like, what's going to work, what's going to not. And so, you know, marketers tend to be experimental by nature. It was like, let's try this campaign and this spend, and let's see if this ends up being a good use of money. The problem ends up being that, like, measuring which things are working ends up being very fraught. You know, as you know, the world of marketing attribution is a famously hard problem. You know, I. My background's in data science. I supported marketing teams at webflow and Airbnb. You know, Michael and I first met because, you know, not just contextual abandonment, but also because at Webflow we could not measure our YouTube ad campaigns. You know, just the thing about YouTube ad campaigns, it looks a lot like TV ads. You know, you don't click on them, you just watch them. And so this world of how do you actually be experimental, where you actually know what works or what doesn't, ends up being pretty fraught. So this is again, a place where you want to get the science right and marketers also know the metrics you want to use are the bottom line, things that are going to serve a payback equation. Things are going to drive revenue. So what we recommend with most marketers is take your kind of riskiest marketing initiatives, which tends to be the largest spend, and figure out how do you be experimental there. And by and large, I think for most teams, that's going to look like performance marketing spend, brand marketing spend. And that's where I'm very excited about this new suite of tools that we're both working on from episode. We're thinking a lot around this incrementality test capability of just what is it? How do you prove out that this channel has roi? And then of course, like, people will need some sort of universal model of, like, how do you bring it all together to kind of, you know, determine where to go?

Incrementality testing, obviously, really important. Really. I think I've been really excited to see a lot more marketers talking about incrementality testing over the last couple of years. I think it's always been part of, you know, some amount of experimentation has always been a part of, of marketing ever since, you know, Hopkins Scientific Advertising from, I don't know, like 1902 or whatever that book was published but like a long time ago. So, like, the ideas have been around. Would love to hear about, like, modern approaches to incrementality testing and sort of like, what y' all are seeing that seems to work and like, where are the opportunities? This is. I ask about it because it's a little bit different than like sort of your A B testing. And so would love to hear, like, what are y' all seeing? What seems to be working? What are you excited about in terms of the landscape?

Yeah, exactly. I think it's an interesting place because I think it's an industry in motion. I think it's similar to where AB testing was in the optimizely days, where people could get that we should divide people into two groups and measure them. I think incrementality as a concept, like, oh, yeah, we should inflect, spend up or down in particular units and then compare them to the Others and that's a way to use both first party data and you know, do it in a way that kind of controls for a lot of things, but the actual mechanics and science of doing it at scale in a trustworthy fashion. I think there's still quite a lot of, you know, gaps that exist on the market where like if you just look at what any experiment is and it doesn't matter whether it's an A B test or inquiry test, whatever else it is, it's got these steps right? First you gotta design the experiment like you know, the same thing as like where like if you just look at what any experiment is and it doesn't matter whether it's an AP test or an inquiry test, whatever else it is, it's got these steps right? First you got to design the experiment the same thing. As you're about to write a paper with some new cancer medicine, you're going to get approved to do it with this many people under these circumstances, measuring this thing, that setup is going to be something you're about to write a paper with some new cancer medicine, you're going to get approved to do it with this many people under these circumstances, measuring this thing, that setup is going to be something that both de risk the business where if there's some sensitive places or important areas, maybe avoid them and it also makes the test viable. If this test has no chance of actually getting signal from noise, then why are we doing it? So it's similar deal as the A B test of getting enough sample size for power analysis for the incrementality test. How do you make sure it's a viable test and how do you make sure it de risks the business, it avoids sensitive areas sort of thing and then you get into implementation. This is where using all the martech tools properly is really important. How do you set up audiences appropriately? How do you do it in a sort of automated fashion? And then when you get into the measurement, I think actually the measuring of an incrementality test is probably on the part that actually is further advanced than others. There's a lot of packages out there that will let you do a lift test. We have our qualms with many of them. We'll be definitely doing our own approach. But it's similar deal if you want to do a T test and a B testing lan turns out you can just, you know, use a package for it and then to actually what is your takeaway? I think that's another place that's kind of interesting where like, you know, you're not randomizing in an incrementality test. And so, you know, the amount of kind of predictability, generalizability that any individual test has is just much lower than an AB test. And a B test itself also has some issues there. So, like, when it comes to incrementality testing, you know, we believe as well that like, you know, it's got to be something that is manifold. You know, you should be ideally, you know, basically like a B tests tend to not get replicated. Incrementals should probably get replicated. You should probably, you know, do more folds of it. So there's a lot more to there. But like, I think that we're excited because I feel like, because we come from the data science world of rigor and because, you know, we come from this A B testing world which understands what it means to make it systemic and continuous and pervasive. I think we're, we're just going to, we're excited to raise the bar on what we're seeing on incrementality tests.

Love it. I'm, I'm really excited. I, for, you know, I mentioned that I am excited that people are talking more about incrementality at all. Like, that is on its own a really good thing and that experimentation is growing as a practice, especially for marketers and especially to like, not just, you know, we're going to test the subject line of the email, but really we're going to think about what's the true incremental value of this activity and how is that driving our business. Bottom line, I think that's amazing. And I really like what you said about sort of incrementality testing being where a B testing was like five or 10 years ago. It's like people are starting to get excited about it, but the tools aren't great yet. And that just means that there's a lot of opportunity to really grow this and make it easier for people to do the right thing and then start to increase that testing velocity. And I agree. I think for incrementality testing you really need to be doing it as much as you can. And because of the way that everything changes constantly in the world of marketing, like, you don't just like, you know, say, like, oh yeah, like we tested YouTube and it worked and we're never going to test it again. It's like, no, you need to be testing it every quarter at least, because it's going to change. Like it's. The things that worked two quarters ago aren't going to work anymore because of the nature of the way that these advertising platforms work. And so having more and better tools to be able to do that, I think is super important. The last topic that I, that I want to hit on today is a thing that I find very frustrating, personally. Like my. When you are working in this space of data science and statistics, there is sort of facts of life that have to do with uncertainty, right? Which is like, we are not, you know, you run the test and there is some amount of lift and there is uncertainty associated with that lift. Business users hate uncertainty. They like, do not want to hear that. They want to know, like, did it work or not? Like, Give me the 1 number to plug into the spreadsheet. How do you deal with that? How do you help people understand uncertainty? Should we do that or should we just give up? Like, how should we think about like, as data scientists and statisticians communicating that uncertainty, helping business users make good decisions in the face of that uncertainty when they often like sort of reject the idea of uncertainty from perspective.

It's such a fun topic. I. We publish this blog post called the the Cult of Stats Sake because it talks about how people were overly emphasizing like, you know, point estimates and P value thresholds and like, you know, people like, they really love these sort of like hard kind of cliffs with things when that's just not really what the science is telling you, you know. So it's, it's definitely a problem we think a lot about. You know, my ideal world is where everyone reports on confidence intervals. So that's the thing you're actually being delivered, right? It's like a rate here. It's somewhere within this range. It's sort of. But like, you know, the human brain unfortunately isn't ready for confidence intervals. So like it's, you know, we're in political season. There's all these polls coming out. All the polls are basically given confidence intervals and no one says anything, right? There's like, it's. No one mentions that. So, you know, I, I'm very much a. When I think about experimentation and probably in France and stuff, you know, I'm very much a realist of like, you have to meet people where they are and how the brains work. The way I see the world is that people are going to report point estimates, but the application of the point estimates, we can provide them tools that do it better. I alluded to the. How do you come up with aggregate impact? It turns out you cannot just sum your point estimates because of this thing called the Witness curse. And in general sort of Generalizability issues. But there are tools that help you aggregate. Ideally you're running a whole group and you do it that way. But if not, the world of Bayesian has much better language for how do you kind of create aggregates out of individual level experiments? Because they have these priors, you have inbuilt shrinkage and you can kind of end up with something that says if you ran your experiment for a short amount of time and it's more uncertain, we're going to apply more shrinkage and we're going to bring it much more to bear into something normal. Whereas if your thing is super proven and data is very, very, very good, then we can trust it much more and we can push it up. So, yeah, what I'd say is like, you know, I, I would love the ability to get people to start talking. Much more confidence intervals. But, you know, observing it across all the other disciplines, no one's cracked that yet. So instead of what we can do is just say, let's just make sure they don't abuse the point estimates where like, you know, as you think about what to, what to do with it, such in agri impacts in particular, like just give them a separate tool for it.

Yeah, I think, I think you have to be realistic about this and sort of assume that people are not going to want to use the confidence bounds.

I'm with you.

I wish that I could just give people confidence bounds and have them intuitive, but human brains do not work that way. It rejects uncertainty. And so I agree. I think that this is actually another sort of part of this world that we operate in that I'm really excited about, which is figuring out how do we give tools to help them, to help users make good decisions in the face of uncertainty. Sort of knowing that you can't just communicate them the uncertainty. And I think there's a lot of like, really interesting work being done. Some of the work that y' all are doing at EPO and some of the work that we're doing at Recast and you know, other, other people are working on this as well. I think that that's a really interesting, interesting direction that the world is heading that I'm. That I'm really excited about. Yeah, well, we are out of time. This has been a wonderful conversation. Thank you so, so much, listeners. Go check out epo. I believe that we have a few links that we are going to be sharing out alongside this conversation. The website is getpo.com Jay, thank you so much for the time, so much for the conversation. Really really dense, really informative. And we will be talking more again soon. I am very sure of it.

Pleasure being here with you, Michael. Good catching up. All right.

Thank you.

Bye, Sam.