Etsy Sr. Data Scientist: On building a strong experimentation culture | Paul Scheetz | Coffee Breaks

Some of what are the major ways that incrementality testing experimentation? What are the ways that it goes wrong?

We had a slide that had a bunch of quotes that we've heard from our stakeholders who were complaining about geotesting. Some of the examples were we spent a ton of money on a test and then the Olympics caused a lot of disruption and now our results make no sense. We tried to run an experiment in the uk, but London is a huge outlier and the data messed up all the results or one that's particularly painful for me. We ran an experiment, but the confidence interval was so wide the results just aren't helpful. It boils down to four primary categories that we've identified.

All right. Paul Sheets is a senior data scientist at Etsy with over a decade of experience in marketing analytics across agencies, startups and large tech companies. His work includes all things marketing measurement. He recently built a dedicated marketing experimentation platform at Etsy 2 to support all of the site advertising and feed teams. As a staunch advocate for Bayesian statistical methods, Paul has pioneered advanced analytical approaches to measuring true incrementality within Etsy's marketing program. In today's episode, we dove in on a variety of technical topics, touching on the potential challenges behind designing rigorous incrementality tests. 5050 random sampling versus other types of sampling that you might use when you're designing geo experiments, your options when dealing with small sample sizes, synthetic controls, how to balance uncertainty with whether a test is statistically significant or not, and how to build a solid experimentation culture. There's a whole lot more in the conversation and Paul and I went down some deep technical rabbit holes that I think y' all will find very interesting, especially if you're on the scientific and statistical side of marketing measurement. All right, Paul, welcome. It's great to have you to start. I would love for you to quickly introduce yourself. Why don't you tell our audience what's your role and what do you do?

Yeah. So great to be here. Appreciate you having me on, but I am a senior data scientist at Etsy and I am responsible for our general marketing measurement practice. I operate a bit of as a center of excellence when it comes to all things experimentation in marketing. I'm responsible for organizing and supporting our incrementality testing along with methodology research and improvements. Currently, I'm also in the midst of a big MTA rebuild which has consumed about the last six and a half months of my life. But in brilliant. It's something. It's something. But I also work in my spare time on A performance marketing analytics experimentation platform. So at Etsy, we have roughly 105 million products that we have on sale that are being sold on the platform at a time. And we don't really have a good way to scale and standardize our performance marketing experimentation, especially in our feeds practice. So we have been building an internal experimentation platform that allows us to better syndicate and then we get feedback on the results of those experiments. My free time, I like to make emojis, ms, paint pictures, and just cause general shenanigans on Slack.

Okay. I mean, I would love to dive into how you. How one makes an emoji, but we don't know where. I want to keep this focused on the marketing science. So that's great. I mean, everything that you do, I think is going to be super relevant for our audience. You know, I've followed your work, blog posts, LinkedIn stuff for a long time, love everything that you do. And you know, recently I had the pleasure of seeing a talk that you gave with one of your colleagues, Vishwa Bhuta, on experimentation, measurement, marketing measurement that I really loved, that I thought was super insightful, super well thought out. And I'm really excited to dig in to some of those topics, specifically around experimentation, because incrementality testing, experimentation and marketing, it's really hot right now. But I think y' all did a really good job of talking about, you know, some of the places where it gets hard. Right. Where it's not just as easy as like the nice, you know, little graph in the, in the blog post. But really there are some, some deep challenges. So, you know, I would love maybe to start if you could talk about some of what are the major ways that that incrementality testing, experimentation, what are the ways that it goes wrong? I think that this was sort of a key theme throughout this talk that you gave. So maybe you could highlight some of those, you know, the big things that practitioners should be looking out for if they are looking to get into incrementality testing and experimentation as part of their marketing program.

Yeah, absolutely. So one of my favorite slides that we put together for that slot for that presentation was we had a slide that had a bunch of quotes that we've heard from our stakeholders who were complaining about geotesting. Some of the examples were we spent a ton of money on a test and then the Olympics caused a lot of disruption and now our results make no sense. Or we tried to run an experiment in the uk, but London is a huge outlier and the data messed up all the results. Or one that's particularly painful for me. We ran an experiment but the confidence interval was so wide the results just aren't helpful. And these, these kind of go on for a little while, but we, these.

Echo things that I have heard also many times talking with practitioners. So this, this feels very familiar.

Yeah, it's, I think it's a pretty universal sentiment really. What can't go wrong with geotests? It seems like there's always something, but really it boils down to four primary categories that we've identified. You have statistical issues, technical issues, generalization issues, and then this, this broad other issues. So when we say statistical issues, the key problem here is that when we're running a geotest we're typically running it with a limited number of GEOs. So Etsy we're running on DMAs and in the US we only have 210 DMAs. Internationally we have fewer. I believe the Germany has around 82, UK has around 76. But essentially geotesting is a small sample size problem, which means you have a longer runtime to power, which means imprecise results and then just limited granularity. It also comes with this fun statistical phenomena of statistical noise. Because we have so few GEOs and experimentation units, any noise in one of those is going to cause problems within your test. And then we also have this small effect size for some of our certain channels. So I think a lot of, lot of businesses have a few channels that are driving the majority of marketing revenue and then several smaller channels. Running a test on these smaller channels is really, really hard because the effect size is just so small you either have to run your test for six months or not be anywhere near confident in your results. From a technical standpoint, it really boils down to the reliability of your targeting. If you go too granular with your tests, the targeting on the ad platforms gets really shaky and you start getting this contamination between your treatment and control. So if you test a zip code level, I think the efficacy from what Google tells US is around 70% reliability in targeting somebody at a specific zip code. So typically we target at these broader regions. And from a generalizability standpoint there's GEOs just aren't the same. New York is not the same as Rapid City, South Dakota and there's non homogeneous treatment effects within these GEOs as well. So you may have a much higher incremental effect in a smaller GEO where you don't have as much penetration versus a larger GEO where you have high competition and high market penetration. And then finally you May just have black swan events. A couple years ago we tried to run a geotest in Germany when they're having all that flooding and we did not get a good read. So those are the broad. And then obviously you have channels you just can't test like SEO. Unless you can convince your stakeholders to delist your, your site entirely, you're just never going to get a real true incremental read on the efficacy of SEO.

Okay, so all of this resonates with me. Lots of, lots of issues. I think you know the core one or so just to restate some of these. So one is just that like small effect sizes. Right. And I think there was a published paper that I'm sure you've read. I'm blanking on the title of it. I was talking about someone recently and it's like the unfortunate economics of experimentation and marketing which is just that like it's very hard to get a well powered test because the effect sizes are so small. Like the effect sizes are small. The impact of like someone seeing one ad on a purchase decision is very, very small. And so getting a read on it when you're, when you're dealing with small effect sizes, getting a read on it is just hard. And so there's that underlying everything that we're doing which is that we should shouldn't expect to be able to get very clisp crisp clean reads on marketing where we expect that the effect sizes are generally small. And then there's all of the other things that you're talking about which is just that we have small sample size that compounds the problem because we're working with geographical units. If you use smaller geographical units, there's other trade offs because you don't necessarily have the ability to target people as effectively as you maybe want because of the bleed over. You target one zip code. But then someone goes, you know, to their neighbor's house and actually gets exposed to the ad when you thought they were part of the control. Another neighbor, but maybe their brother in law's house right across town. And so you have those problems and then you have you know, other just like random events that can wreak havoc on, on, on your. Not really wreaking havoc on your data but it just makes the comparison like you can't just do the pre and post comparison because like what if there were floods? So all of those problems I think resonate and you sort of started to talk about some of the ways forward or how you think about adapting to this, you know, unfortunate world that we live in where we, where we can't, there's no easy answers. Right. But so maybe you can start talking about like how do you think about designing experiments that give us at least some amount of signal in this world where everything is, is hard for all of the reasons it's just mentioned.

So I think, I think you're not going to want to hear this but be ready to spend lots of money. So geotesting is something that in a sense you can throw money at the problem and make it go away. But finance typically doesn't like that. I mean I, I see that kind of tongue in cheek but for those small channels you can run them as a heavy up test where you spend more into that channel. But the issue, you will get a read from that. But the issue with it is you start running into, I mean more confounding factors like you might be oversaturating the market there or you might just have diminishing marginal returns as you spend up. So you can get a read by spending more money. You can increase the power of the test by potentially limiting, cutting back on spend in GEOs as well. But I mean again I guess it's a lot of the times these are decisions that finance doesn't really like. But I guess my overall advice would be if you're using a 50, 50 random sample split, don't do it anymore. There are better options out there. I would recommend synthetic control or matched markets. We typically use matched markets at Etsy but we've also experimented with things like stratified sampling or matched pairs with like a trimming process.

But okay, let's, let's dig in, let's take, let's dig into that because I think, I think not everyone who's listening is going to have fully understand what all of those words mean. So yeah, let's start 50, 50 random sampling. What is the appeal of that? Like just describe what that is and then let's talk about the drawbacks and then let's talk about some of these other methods that you, that you just talked about.

Sure. So 50, 50 random sampling, you take your pool of geos, you split them randomly into half, get the treatment, half get the control. The issue here is that you can really, really easily skew your sample size. So let's just use the United States as an example. Etsy, some of the major markets disproportionately drive quite a bit of a revenue. So New York, Louisiana I believe Dallas and Chicago are another big ones. If you happen to randomly sample and you have most of these markets in your control or your treatment group, small changes in those markets are going to cause really, really big effect sizes which potentially don't generalize well to the rest of the market once you roll that test out. So you just have to be really, really careful that your random sample in the small sample size is actually representative of everything.

So I think one way that I would restate this or one way that I, that I think this makes sense to me. So random sampling gives you nice performance under repeated, repeated trials, right? And so random sampling is great, but you only really get the true benefits of the random sampling if you have very large in size such that the randomness gives you a bunch of guarantees that the treatment and the control are going to be comparable or you repeat the trial multiple different times. So you can do 100 different trials and all of them are going to have random samples. And you know that, you know, some are going to be weighted towards the treatment and some might be overweighted to the control. But, but over all of those different trials you get the benefit of the random sampling. But if you're only doing one trial, which is generally the case that we're talking about in marketing experimentation, you don't get those guarantees, right? One trial with only, whatever it is, 150 DMAs, you could very easily end up with 75. Like your 75 treatment DMAs just randomly ended up all being on the east coast and your 75 random controls all ended up being on the west coast. And now all of a sudden these aren't really comparable treatment and control. And so the point is that the simple random sampling actually in small incisors doesn't give you guarantees of comparability between treatment and control. And so you need to be a little bit more careful than that. And so that's one point and I think some of the you mentioned some tools for dealing with that like stratified sampling or match markets. And so I want to talk about how those actually work. And the other point that you mentioned that I think is really important is this idea of like national representativeness which I think actually a lot of the talk around experimentation recently hasn't focused on this enough, which is that really what we're doing, right? The whole point of doing a geolift type experiment is try to learn about, okay, how does this marketing program work when we run it in a business as usual basis, which generally means some like national campaigns. And so implicitly what we want as part of a feature of our experiment is that when we run the experiment, those results that we get on the other side, are going to generalize to that national campaign that we use for planning purposes. And so if we only ran our experiment where we only ran the experiment like the only treatment geographies were like Houston and Nashville or something, there's not a lot of guarantee that that's actually nationally representative. And so if you have smaller numbers of treatment geographies or even if your, you know, simple random sample isn't, doesn't end up being nationally representative, then you end up in this world where it's unclear how well those experimental results actually generalize to this national campaign that we care about. Is that a good summary of sort of like the different items that we're talking about here?

That is a really good summary and I wish I was that articulate because I'm going to steal that and share that with myself.

No, I, I got the benefit of listening to you talk about it and then I was able to easily summarize. Okay, so that's the world that we're working in. We've got the problem of national representativeness and we've got the problem of like random sampling not giving us a lot of strong guarantees in the problem of not giving us giving us strong guarantees around like compare the comparableness of treatment and control. So let's talk about the second thing first. I want to come back to nationally representativeness because I think that's an interesting one. But with the problem of small sample size, we want treatment and control to be comparable. What are the options that we have for doing this that isn't simple random sampling? That has better performance than simple random sampling?

Yeah, so I, I mentioned we, we at Etsy use matched markets. So this is again a non random sampling, but you're essentially creating two subgroups of GEOs, treatment and control subgroups that statistically represent one another well. And when I say statistically represent, there's a litany of validation tests that we run this against to make sure that it's robust and the relationship we can make sure is stable.

Wait, so maybe let's dig in there. So what are some aspects of that that you might look at to say like okay, yes, this has these properties that, that we. Or like what are those properties that we.

Yeah, so I mean obviously correlation between the two aggregated groups is a big one. We test for structural breaks. So if there's a structural break within the one of the two series then we have a little bit more maybe doubt, get doubt around it. Out of scope test is a big one that we've recently implemented. So what we really want to avoid here is we don't want to get into like a spurious correlation where solar power cheddar cheese consumption is correlated with solar power generated in Haiti. We really want to avoid that. And so we'd run this out of scope test where it's essentially a placebo test where we're running a fake geotest on a period where nothing should be happening. And so we should see an effect size of zero during that. And so if we do great, unfortunately a lot of the times we actually see a false effect there. So that's, that also is something that we use to rule out specific combinations of these geos.

Got it. So the idea is that what we're looking for is we're looking for geographies that are highly correlated in the time when we aren't running an experiment and then running some checks to validate that like there aren't these like spurious breaks or things that would give us a false positive when we are actually running experiment. At least when we're looking at historical data.

Exactly. And the correlation is directly, it directly influences the end precision that you're going to get from retest. So higher correlation between the two GEOs or the two groups of GEOs within your pre test period means a higher precision when you actually go to analyze the test.

So how are you choosing these match markets? Is it like look, I'm going to, you know, take this list of DNAs and then sort of eyeball it and be like, I think like LA should be similar to San Francisco. So that's a match and here's this other one and that's a match. Or like do you have an algorithm for this? Or like what is the. How do you power this?

Yeah, so we've started using, Google has a very nice package called time based regression match market. But essentially it allows you to provide a list of GEOs. You can classify them as eligible for treatment, eligible for control or exclude them altogether. And then there's a series of constraints that you can introduce here. So aside from all the statistical considerations that we use when we're selecting these geos, there's also a bunch of business constraints that we're interested in. Typically we want to make sure that we have a minimum amount of revenue coverage in our treatment groups. So we don't want our treatment group to be so small that it only represents maybe 1 or 2% of our total total revenue. Again it's kind of for generalizability purposes. There's also other business considerations where we may be running other tests or other media and other markets where we just want to exclude them altogether. So again we're introducing these constraints into this both statistically and business related. And the nice thing about match markets is it allows us to do a lot of that. But occasionally there's, you have to be a little creative with how you, how you introduce those constraints where you have to modify the underlying objective function. But from a technical standpoint what matched markets is doing is it's using a greedy hill climbing algorithm. So this, this idea of selecting a control group and a treatment group is actually, it's a really, really hard problem. It's from a computational complexity standpoint it's NP hard. So if you can figure out how to solve this in quickly, there's, there's a million dollars waiting out there for you. But. Nobel prize, Nobel Prize.

Do you have a solution to this? You can come work at recast we beer from you.

Yes, yes please, please let us know. But yeah, it's, I mean there's, there's sort of these, there's a bunch of pseudo algorithms that you can use to get close enough solutions here. Worst case scenario you can brute force it, but that if you start using too many GEOs that can take heat from now essentially till the heat depth of the universe to come up with enough combinations. But yeah, typically the matched markets is what we're using the TBR one.

Got it. Okay, great. So that's helping, you know, solve some of these problems under the hood to try to figure out what are, what are a good set of match markets. So match markets I think makes intuitive sense.

Right, right.

To people it'll be like okay, we'll have some treatment markets and then we'll choose the best matches for them. You also mentioned stratified random sampling and my background is in econometrics and so I love the idea of stratified random sampling. Why don't you explain like what is stratified random sampling? How does that work?

With stratified random sampling essentially you are creating, you're sampling from strata which I guess just a really, really simple example, like let's just bucket all of our GEOs into high, medium and low. So if you're a high, high revenue GEOs, so New York, Louisiana, Dallas, Chicago, you have your, a tranche of medium revenue GEOs in the atranche with low revenue GEOs. And essentially what you do is you randomly sample from each tranche. So 50% of your high, high high revenue tranche would go into your treatment group. 50% of your medium tranche would go into your treatment and then 50%. So instead of just doing pure random sampling across the board, where you have everything, you randomly sampling essentially at the block, at a block level.

Great. And so the, the idea behind this is that this can give you better results than simple random sampling, especially in low in size situations, because you can sort of force the, the end state to be reasonably representative, at least over, you know, the characteristics that we think might influence the outcome. And so in the healthcare world that I used to work in, like, stratified random sampling often makes sense. You know, you would strat before sampling, especially under small insize, you maybe stratify on gender. Right, because males and females might have different responses to some treatment and you don't want to actually end up in a world where like, oh, all of our treatment sample is, you know, 80% men and the control sample is 80% women, and therefore they're not like, actually comparable. And so we stratify on gender and you stratify on a couple of other important characteristics in order to make sure that your sample is going to be balanced at least across the characteristics that we have some reason to believe are going to impact, you know, the relationship between the treatment and the outcome. And so, so in geographies, you could imagine stratifying exactly as you say, on like something like penetration, on revenue, even like, you know, part of the world. You could have like a northeast strata, you could have a southwest strata, and then sample within those to give you more like guarantee yourself more geographic coverage between your treatment and control group than you would be able to guarantee with just, with just straight, simple random sampling.

Yeah, exactly.

Swap. Okay, awesome. And then the last thing that I want to talk about, and again, because a lot of people have been talking about this recently, is the use of synthetic controls. So we've, you know, synthetic controls. There's a bunch of things that we could talk about, but I'd love to hear, like, how does synthetic controls fit into this picture as we're thinking about comparing it with especially match market testing or stratified sampling.

I wish I knew more about synthetic controls. I've used them in the past, but I would not claim to be an expert in them. But synthetic controls essentially allows you to create. So with matched markets, the way that I like to think about match markets, it's essentially a synthetic control with a binary weight, more of a traditional synthetic control allows you to have partial weights. So going back to my example of New York and Rapid City, New York might be equivalent under, under synthetic control paradigm, New York might be equivalent to 0.5 Chicago plus 0.5 Miami plus 0.2 Rapid City. And again it's, it's essentially a linear regression. Right? So yeah, you're just getting those partial weights from those markets and you're using those to create essentially a synthetic version of New York for your counterfactual.

Yeah, so I think that makes perfect sense. I think the way that I would restate this is like New York doesn't have a perfect match market in the United States. Like there is no market like New York in some very meaningful ways. Like New York is special. And so you know, in a match market algorithm, maybe the best match for New York is Boston. But anyone from either Boston or New York would very quickly tell you that like that's not actually a good match. Right. For a bunch of different reasons. And so what synthetic control allows you to do is it allows you to create, instead of having to choose one market that is the match for New York, you choose a weighted combination of multiple different markets. So exactly as you're saying, maybe like the best fit for New York is like 30% Chicago and 40, 40% Boston and 20% something else. And that actually gives you a better match to New York than any one of those cities individual or any one of those DMAs or states or cities individual. Is that like a. Right, the correct summary of this?

Yeah, I think that's, I think that's spot on. The other nice thing about synthetic controls is you can start introducing more sophisticated machine learning type approaches where you can start incorporating regularization and just a more sophisticated matching processes compared to just straight one to one matching.

Love it. Great. And the idea behind all of this, right, like the whole reason to do it is to hopefully reduce the noise, right. Because if you have better matches then you have, you know, better pre treatment match and therefore you get more signal. In the post treatment world of being able to determine like how much of, you know, the change in, you know, revenue or whatever the outcome is is due to the treatment versus could just be random noise. And so that's the whole point of doing all of this and sort of thinking about like noise and lack of signal. One of the things that you mentioned at the top of the call was that stakeholders get frustrated where they're like the uncertainty interval of this test is too wide, like what do I do about it? And I would love to dig into that because I think it is sort of on the one hand a thing that everyone who works in this space like has heard and has felt, but also has some like deep, sort of profound way, really deep aspects to it in terms of like, what can we do about it, what actions can we take? How do we think about structuring our experiments? So I'd love to dig into that. Like when someone comes to you and says, oh, the uncertainty bounds on this are really wide, or even says, hey, the results weren't statistically significant, what does that mean? What is your answer? How do you, how do you think about that in a business context?

Ah, yikes. So that's a, that's, that's always my worst nightmare when we come back and the test is not statistically, statistically significant or we just have so much noise that the results are essentially unusable. Um, there's a couple of ways that we've approached this in the past. One of the ones that we initially started with was essentially bootstrapping our confidence interval. So under, unless you're using a Bayesian paradigm, you're not really getting a whole lot of information around where, what the confidence interval actually looks like. But with bootstrapping you can start to see more of the shape of it. And I know it's kind. But what we end up doing is we run effectively repeated trials within our treatment and our control group. So we'll select a subset of GEOs within our treatment group and then we'll find the group within the control group and the excluded GEOs that best represent that. We'll run it through that same litany of constraints and then we'll get the IROI for that specific subset of geos. And we do that a whole bunch of times and then eventually we'll start getting the shape of what we believe the true distribution of the confidence interval looks like. And by doing that we can oftentimes see that we have maybe a skewed distribution. And so we think that the effect size is actually somewhere within this higher density area versus kind of this longer tail where maybe the, the actual readout or excuse me, where the confidence interval is suggesting. So that's one way we've approached it. We've also approached it from, I guess the way that I would describe this is a what would we need to believe for this to be true type of framework where if we see that essentially we use that to narrow the confidence interval, we run essentially a sensitivity analysis. So if we say that our Iroi for Google Shopping is 0.5 versus the 1.6 we were expecting, we would need to believe that the overall business accelerated by 10% within the experimentation window for this 0.5 to actually make sense with the data that we were seeing. And so we're able to kind of narrow it down that way by seeing what we think is actually possible from a business standpoint. And that's been a really, really useful tool that we've worked with our stakeholders on from a technical standpoint. There's a couple other things that you can do. The big one that we've started doing lately is introducing Cupid. So if you're not familiar, I need to actually remember what this stands for. But Cupid is controlled experiment using pre experiment data. And the way that I like to describe Cupid is it's a bunch of computer scientists who essentially rediscovered introducing control variables into a regression equation. But what it allows you to do is it allows you to introduce other variables that are uncorrelated with your treatment status and use that to control for maybe some of the pre existing variation in your data. So reducing the noise. And there's, there's extensions to that. There's another one called qpack which is, I like to describe it as a fancy dimensionality reduction exercise where you're taking all these covariates, mashing it into a machine learning model, taking the output of that and then using that as a Cupid variable. So that's kind of useful because you can introduce all several more variables versus just straight Cupid. Because with straight Cupid you're going to run into a P is greater than N, N is greater than. Can never remember which direction that goes. You're, you're, you're going to run into the curse of dimensionality pretty quick with Cupid if you start to introduce multiple control variables.

Okay, great. So again, just to sort of summarize some of these different approaches, there's one bootstrapping to try to understand the like shape of the sort of distribution of the expected values that are compatible with the test and then see if there's a particular high density area because the, see like you know, just 95% CI that might pop out of a regression might actually like not tell the full story on where the density of values lies. So that's like a good exercise to go through. And for the listeners be careful with using the bootstrap method because that is a, it is a, a gateway drug to Bayesian statistics. So I'll say be very careful. That was how I got into Bayesian statistics with, by really using and abusing the bootstrap. So FYI, okay, so that's one way other things you mentioned qped and qpack, which is basically just doing like control variables in terms of the analysis of the test that can help to reduce noise and therefore tighten the confidence bounds a bit. And then I want to talk about the other thing that you mentioned, which was the what do we have to believe? Analysis. Because I think that that's actually maybe the most practical thing that people should be doing also to some extent the hardest, because it's not like, oh yeah, we'll just like run some, you know, algorithm in Python or an R. But really it like requires thought. And so it sounds like what you're describing is you're, there's sort of a couple of different dimensions of this. There's first of all, like, we got these results from the experiment and now we want to think about like, do we actually believe these results or not? In order to believe these results, we need to sort of think about that in the context of the rest of the business, like, in order for these results to be true, that implies some other things and we can say like, do we believe those other things or not? And I think that, that, you know, is one route and maybe another slightly different version of this that I often recommend is to think about is sort of flip the question on its head and say like, okay, given these results, what do we do about it and to what extent do these results change or not the actual business decision that we need to go make? And I think those are both like two really important aspects of actually running an experimentation program in practice is, you know, are these results consistent with other things that we believe or not and should that cause us to like, down weight how we use these results in our decision making process? And the second is actually really focus on the decision making process, which is to say, like, there are some classes of results of some experiment, like, oh, we have really wide confidence bounds, but actually it doesn't really change what the right next step is. Right, which is, you know, pulling back spend on this channel or doubling down investment into this channel. And it's actually like the decision making thing, the decision making process that really matters. And so we should always be thinking about like, statistical significance doesn't necessarily matter that much. What actually matters is how does this, how do the results of this test impact the actual, like, go forward decision that we need to make as a business?

You're sending more and more liquidations. Statistician.

Oh no, did I give myself away?

You might have just given yourself away there with the, the focus on the business outcome and the decision making process.

But I Do.

Go ahead, go ahead. Oh, I was gonna say, I think you're spot on with both those sentiments there.

So the last thing that I want to dig into, and this is a, this is a pet peeve of mine, is that I talk to marketers who are running their experimentation program and they, you know, keep expressing this frustration with getting non significant results and they're like, yeah, we ran this experiment, the results weren't statistically significant and so we feel like we're failures. And I, I think that that's actually like just wrong in a couple of different, in a couple of different meaningful senses. So in the world of marketing. Well, a couple of things I guess to, to clarify and so I'm going to sort of explain why this annoys me and then I would love to get your sort of comments and thoughts on this. So this annoys me. One, because statistical significance has a very particular meaning to statisticians. And I think marketers unfortunately think that like, statistical significance means good. And so like, they're like, oh, if we didn't get statistical significance, it means that the experiment wasn't good and we failed. But really, like again, when you're interpreting statistical significance in most contexts and in most cases, you know, tools that you're using or reports that you're looking at, what people are showing you is whether or not the results are statistically significantly different from zero. And you can have an experimental result that is still very useful to you as a business that does not yield significant results that are results that are statistically significantly different from zero. In fact, in many marketing programs you would expect that in many cases you could run a great, totally valid experiment that does not yield results that are statistically significantly different from zero, because that channel or that tactic or whatever you're testing just isn't that effective. And so really what I want to push marketers to do is to not like panic about whether or not results are statistically significantly different from zero, rather focus again on what do you actually need to know as a business. And it might be the case that you ran an experiment where the results weren't statistically significantly different from 0, but the incremental ROI is statistically significantly different from 3 or from 5 or from some other number that actually does allow you to make a good decision. You could say, look, the results aren't compatible with this, with this channel having at least a 3x ROI and we know we need a 3x ROI to be profitable. And so actually that's a good result for us as a business. I mean, it means the channel isn't working very well, which might be bad for, like, you personally as a marketer, but the decision to take as a business is actually very straightforward, which is like, we should reduce investment into that channel, because it doesn't seem to be driving very much effect for us. So, I don't know. That was sort of like a long rant. But this is a thing that I. That I see a lot that I'm curious to get your take on it. Like, is that a thing that you see? How do you think about this difference between statistical significance that a lot of people really hone in on, and, like, the rest of the information from the experiment about what do we actually go do with it?

Yeah, I'll. I'll add one other.