Amazon’s marketing incrementality & experimentation approach | Ben Skrainka (Amazon) | Coffee Breaks

You've mentioned this idea of brand lift now a couple of times. It's, you know, really, really difficult to measure what this long term effect of the value of a brand or the value of quote unquote brand advertising. What is the right way to think about this? What's the right way to measure it?

It's a super hard problem. And a lot of the beliefs in the industry aren't based on causal studies, they're anecdotal. There's not something like a matching estimator or any attempt to make this causal and credible. Also what makes this hard is we don't fully understand and we don't observe the cognitive pathways that go into building mental availability in your mind.

Ben Skrinka is the principal economist on Amazon's Marketing Incrementality and decision support team where he leads initiatives to enhance marketing measurement at scale. His industry impact includes catalyzing and experimentation Renaissance ebay, optimizing AWS forecasting systems that save 50 to $100 million annually and building sophisticated machine learning solutions for companies like Convoy's Digital Freight Marketplace. In today's adding AWS forecasting systems that save 50 to $100 million annually and building sophisticated machine learning solutions for companies like Convoy's Digital Freight Marketplace. In today's episode we dove into the multimillion dollar measurement questions and that economists at Amazon are working. Correctness in science and the replication crisis, common errors when interpreting high velocity tests, the Bayesian case for experiments, how to communicate the value of knowledge to leadership. Can you really measure brand lift at all? And how to make decisions in a world of uncertainty and much, much more. Let's jump in. Ben, great to have you starting off. You're an economist at Amazon. What does that mean? Economists are normally like academics. You know, I think about publishing journal articles, but you're working in industry. How do you think about what your job is as an economist working in the private sector?

Yeah, Michael, that's a great question. And so for me it means something different than someone who's just joined Amazon. And this is actually my second time at Amazon and when I first joined Amazon Back in 2012, there actually wasn't an economist job family. And so it was kind of a dire thing. I came out of academia and they dropped me on a floor and it's like endless seas of bartlebly. This Gribner with MBAs and you know, they come and ask me these questions, I'd be like, that's per se illegal. Like don't ever send me an email about that. So What I do and what's so amazing about working at Amazon is that we use economic methods and theory to try to tackle really hard problems like measuring, marketing and. Right, that's really easy to say. But as you know, this is a super, super hard problem and there are so many layers to it, whether it's experimentation or observational. But there are economists in all parts of Amazon and they've had a huge part role in Amazon success. Everything from, you know, how do we think about labor market stuff, people and compensation and incentives to forecasting for capacity planning, which is something I did in a past life at Amazon, which is another one of these multi billion dollar questions to pricing and inventory. And a lot of people may also not know this is that economics and operations research used to be very closely allied field. So there's a lot of overlap there around optimization with scarcity, thinking about supply chain. And now also we have all these public policy questions given I think the polite term is the macroeconomic uncertainty in the policy space that we have to think through. So for me, part of the reason I came back to Amazon is I don't know of any company that really is a better place for an economist to work. In industry we work on problems that are every bit as challenging as academic problems, but they're profoundly relevant. We're trying to solve relevant real world problems. We may not be trying to solve them to the standard of a journal article because we're trying to move quickly and at the speed of business. So like I said, Amazon provides great support for economists. As an entry level economist, you spend a lot of time probably in the data building models, running regressions and as you move up, you project influence at a larger and larger scale across the organization. So today, which was not a typical day, began at 8:30 in the morning with a meeting with SVPS where we were talking about some really ambitious marketing measurement questions. And so I spent a lot of time also with stakeholders. I spent a lot of time mentoring junior scientists to help them be more effective. And on a really fortunate day I get to spend time with data and models and stuff like that. So it's a wide range of things which I enjoy and I'm sure that was a super long answer. But it's really many, many things and that's part of the excitement. But at the end of the day we're using economic methods to help people make better decisions using data.

That is amazing. It sounds like a dream job for a lot of economists. What is the difference between like an economist and a data scientist? Because like a lot of the things that, some of the things, at least that you talked about, building models, running regressions, presenting analyses, like that sounds like what in other companies might be a data science role. Like, how do you think about what distinguishes those different roles or how do you conceptualize the difference?

Yeah, that's a great question. And I've worked as a data science and I've worked as a machine learning engineer at different points. Between my first time at Amazon and my second point, there's a period where I actually train data scientists at Galvanize. So the first thing I'll say is I don't think that data scientist is a very good job title because it really doesn't tell you anything about what the person does. Like, if you look at someone's job title and they're an economist, you'll know that they have a specific skill set or at least as an economist. And like, you know, if I meet a macroeconomist, I know they're going to have a different set of skills than someone who's a micro economist. So a data scientist is basically someone who's got serious quantitative skills. They can write some code, they can do some stuff with databases, and they can build scientific models. But exactly what form that is going to take, it's really hard to tell. They could be very focused on deep learning and great at building complex deep learning models and training those systems. They might have come out of physics or they could have come from statistics and have a very much more of a machine learning, traditional machine learning view of the world. And they're very focused on predictive problems. So data science is a very, very broad term. And I think it's really, we have data scientists on our team and one of the things that I think I've learned in many years of working in science is the more diverse my team is, the better the decisions we're going to make. We're going to have more diverse set of skills, more ways to tackle the problem. And that also applies to all that DEI stuff that's out of favor. If I just have white males on my team, I'm going to be like Apple and I'm going to build the iPhone. And I got my first iPhone when I lived in Chicago. I go outside in the winter, it stops working. Right. It was built by people who live in California. Right. It never occurred that like, oh, I'm going to go outside, it's going to be below freezing by a lot and my phone's not going to work, which, you know, you know they should have been able to figure that out. Anyway, so data science test is a broad term. We have data scientists on our team. They come from different, typically quantitative science backgrounds and they have some amount of machine learning and deep learning and database. And I like to joke that if you're mediocre at all these things, you're a good data scientist.

Unfortunately, that might describe me, but I want to change the topic. So we talked a little bit before this call about things that we're passionate about. And one of the things you mentioned was correctness in science, which is a thing that I got really, really interested in when I was working as an economics consultant and I had some questions about the paths to the analyses that we were doing and the correct, the underlying correctness of those results. And I started getting really interested in different ways to think about correctness. And that was all happening around the same time as the so called replication crisis was happening in academia. And I'm curious, you know, given your experience as an academic economist and your work in industry, can you maybe talk a little bit about what is the replication crisis? And then how does that apply to the work, like to practitioners who are, you know, doing economics and doing science in industry?

Yeah, there's a huge, huge conversation we could have about this and it would fill many podcasts and there are lots of journal articles. So this, the simplest version of the problem is like, if you were to go and do like meta analysis of the research literature, you'll see a spike, you'll see like a dip of statistically significant results right below the threshold for statistical significance. So say 95%, you know, papers that are slightly less than that will be very rare. And once you cross that threshold, there's this huge spike. So like, you know, there's this thing that looks like this and you know, like a heartbeat almost on an ekg. And so you're like, you see that and you're like, hey, there's something weird going here. And so if you think also about incentives, this is something we think a lot about as economists. There's this thing called the principal agent problem, which is when I want to delegate to a principal and I want them to behave like me. One of the things I can care about is, you know, the incentives so that they'll behave in my interest. But the incentives in academia in particular are publishing papers and to publish results that are statistically significant. Historically, academia has not put the same weight towards things like tenure that affect researchers on results that are not statistically significant. So if you show that Do a great study and the results aren't statistically significant, then that can be bad for your career. And, and so there are all these meta studies that have shown people are gaming the system. You know, there are famous cases where people's results aren't reproducible. They're doing regression fishing. And so it makes us hard to know how to trust and to move science forward. Also, one of the, I think really interesting things I've learned from running a lot of online experiments, I know with feature experiments, typically only 1 out of 10 to maybe 1 out of 3 ideas will produce an a B test where you can reject the null hypothesis. So once you know that only like 1 out of 3 to 1 out of 10 times, you're going to get a statistically significant result. Like if you see people generating lots of statistically significant results, you should be concerned. And a good example of this was I was at a conference the other day and someone was saying, hey, we ran a whole bunch of a B test and 50% of the time they were statistically significant. We could ship. And so I know something fishy's gotta be going on there. So I think it's really important for us as scientists to do research that's trustworthy. Like when this is your reputation, you know, when you say something is a certain way, you want to be able to take that to the bank that people can trust that. And if you are engaged in what a former manager of mine calls statistical theater, that's worrying. And so like it's really important to be able to reproduce these results. And also, just as in academia, we're not rewarded for reproducing things. In industry it's very rare to reproduce things. Right. If I have some, if I test something, say, I was so fortunate as to run a long term experiment to measure the impact of brand marketing. I found some company that's crazy enough to let me turn off brand marketing for a year or two and then I get this result and I found out brand marketing didn't work. They're probably not going to let me run that again to make sure it wasn't a fluke.

Yeah, so one of the things that you mentioned is really interesting. So you mentioned this idea that like, okay, only one out of ten experiments should be statistically significant. But wouldn't you expect that like 1 out of 20, even under the null hypothesis would be statistically significant at the 5% level, like 5% of the time? How do you think about that where like you're running so many experiments, especially if only 1 out of 10 really should be statistically significant. Like how do you think about the error rates and communicating that to the rest of the organization when you're going to start to get like lots of false positives under a high velocity testing scheme?

Right, and that's a really, really great question. And I just had a conversation about this with a marketing manager. So first of all is, you're absolutely right. This is the case. This is the famous, like, I'm going to go test 40 shades of blue, or in this case, I'm going to test 20 shades of blue. One of them is going to look good, even if blue button color has no impact on outcome. And we see this a lot of time too, when we interview junior data scientists. I sliced the data until it confessed and I found this rare segment where there's positive lift. These were women with blue hair and corgis who live north of the Mason Dixon line. And so we should target this segment. So there are a couple things you can do. The first is there's this paper that's pretty well known from Airbnb where they talk about how to compute a correction factor.

So.

So you want to correct for the fact that my expected lift conditional on it being statistically significant. If you assume your results are normally distributed, you can compute a correction factor. So you should subtract a little thing off and know that there's likely there's some upward bias when you compute the mean of your results that are statistically significant. A more principled thing to do would be to go to a Bayesian setup, if you can. I mean, in general, I think you should use the best tool for the job. And I don't have a strong religious belief for Bayesian or frequentist methods, but when it comes to experiments, I think the Bayesian case is very, very compelling because I can include prior information based on past experiments and other information. The prior also gives me some regularization properties as well. So there are a lot of benefits to being an abasion setup. Also, it's important to remember that one experiment is not a standalone thing, it's a piece of evidence. And that in science we proceed inductively, not deductively. Right? I can't take some economic theory and derive what I think the marketing lift should be. The way I could say in physics, where if I work from conservation of energy here in this, if I'm in this pauper world, I can either reject or fail to reject something so. Or so I can, sorry, I can disprove something or I can fail to disprove something. And so I assemble evidence and if the preponderance of evidence suggests a brand marketing works and I have enough evidence, even though it's all maybe kind of weak, I might say, hey, I think brand marketing actually does work. And but that individual RCT which may be very noisy and have low signal is not a standalone thing that you should make a decision based on. And so it's good to have a pool of experiments and then be able to do meta analysis. Then you'll have more confidence.

One thing that I've experienced and I've talked to a lot of organizations that have experienced something like this, so I'm going to sort of tell a little story and then I'm going to curious to get your reaction to it and potentially your advice. So talk to a lot of organizations where marketing teams and product managers, they're running lots of experiments, right? They've all been trained to run lots of AB tests to iterate towards better marketing, creative and better products, whatever. And you know, they run these experiments and they get, you know, the results that have a p value less than.05 and they say, okay, look, you know, this creative or this change gave us a lift of 1% in the click through rate or 2% on some metric and then they share out these results to the organization. The organization says great, they plug that into the planning, forecast, spreadsheet and somehow the things never come true. It seems like, okay, yeah, we got all of these wins in the product and our creative and if all of these things came true, we should be making $20 million more dollars next year. But they end up not making those 20 million more dollars. I think this is a really common thing that I've seen across a lot of different organizations. I'm curious to get your sort of take on what's going on. Like what's the underlying mechanism here and what should these organizations be doing differently when it comes to running the experiments or interpreting the experiments or using the results?

Yeah, I think these are hard questions and also like you think that these, these different pieces are not independent as part of it. And so if I treat each thing as independent, I mean I've seen this a lot with candidates resumes where you know, if you add up all the lift from everything they did, it's like 50% of you know, the company's revenue. And I asked someone about this in an interview once and he said, oh, it's resume math, don't worry about it. So like you shouldn't be doing resume math. But so in this case, it's hard to. One of the things the marketing people would tell you is like, oh, if you're turning off just that one channel, well, it all works together in a holistic way. And I think, you know, part of this is there may be problems with external validity in some of these studies. That's pretty common or, or trying to generalize from some small study to company wide because they only ran it on a subset or they only ran it on the US and they didn't run it worldwide. And you have markets at different levels of development. And without knowing more about the exact operating environment, it's hard to say, but there's typically interactions or spillovers. A good example of this. One thing I really worry about are, I realize it's a little bit of a degree synthetic control experiments because everyone at a large company like Amazon, we run lots of synthetic control experiments. We're all using the same algorithm. We probably all have the same treatment and control group for all these experiments. So like someone else is running like high overlap on your experimental design. And so particularly big companies, there could be interactions that you just don't know about.

Well, so one of the things that jumps to my mind, and this is sort of teeing up, my next question is that I think a lot, a lot of people who are not trained in like statistics or economics, they have been taught this idea of statistical significance. And my impression is that a lot of what they believe is like, oh, if it's statistically significant, it means it's real, whatever. That's sort of not very well defined because I don't think it's very well defined in their own minds. And so I think, you know, they say like, oh, there's a 5% lift and it's statistically significant and that means the 5% lift is real. And I think that's a problem, I mean, for a lot of reasons. But like one core one is that the statistical significance check under again, you know, in all of these platforms, right, that they might be using is really just checking is it statistically significant different from zero. Right. And so a lot of them, what they're doing is they're getting like a P value of point, you know, 0499991 with some lift. And they're like, okay, great, we can lock in the 5% lift. But really what the, you know, what the results of this experiment have shown is that, you know, the lift is compatible with sort of any number between 0.0001 and a 10% left. And they're not really taking into account that full range of uncertainty and really what the experiment is actually saying. And so I wanted to ask you this question about like P values significance, like when should it should these tools be used, especially by like practitioners, marketers, product managers who are running these experiments in practice and industry?

Yeah, these are great questions. And this also goes back to the whole replication crisis we were talking about before. So the first place to start is if you haven't seen the new paper that came out at the end of 2024 by McShane at all, there's that paper is now the guidelines for the American Marketing association for all their journals. And they had, I think, a five point recommendation around how to interpret P values and quote research. And so first of all, they really want to deemphasize P values. We should, you know, and this is a. They are, we're using a frequentist setup for your confidence interval. You should view the points in the confidence interval as possible value, so you shouldn't anchor on the point estimate. So in particular, this is something we think a lot about here at Amazon is how do I present uncertainty and risk to my stakeholder? And we as scientists have to do a good job of saying that just because your point estimate was, you know, 0.05%, it doesn't mean that the true value is that. And it may have variation from day to day and campaign to campaign because, you know, some days you're brilliant, creative, may not be creative. And the other thing is understanding like what happens at the boundaries because this is where as scientists we have the most pain. It's like your result is, you know, you miss statistical significance by some minute amount and then you find yourself in a room again with a product manager drawing the two bell curves and trying to explain statistical significance and, you know, inference, frequentist inference. And you're like, if I had a dollar for every time I did this, I'd be rich. And, and I'm doing it again and I've actually done it with this person before, it's just not sticking. And they're like, you know, I know you got like a p value of 6%, but that's really close to 5%. Can we just call it good? And you're like, no, you can't because of. And then you lose them. And again, this is a place where the Bayesian setup is better because now you can talk about probabilities and risk. The other thing is there's a working paper that's not well known by Goldberg and John Drow about the optimal significance level to use. And it was done on ebay data in a Bayesian decision framework setup. And you might think that in our world, since we're not designing medical procedures and drugs, that 5% is too conservative. And so many people in industry use 10% significance level. And this paper I think argues for something even bigger than 10% because we're really just trying to place winning bets. And so I think about this kind of in this portfolio problem, right? I want to win as often as possible. And so as I continue to digress on the subject, the last thing I'd say is that there's a paper out of Microsoft about what's your optimal experimentation strategy based on whether or not you have fat tails or thin tails. And if you have thin tails, you should run a few high precision experiments because progress comes from very small accumulation of marginal gains. But if you have a fat tailed innovation process, you want to run a lot of scrappy low cost experiments because you need to find the big winners because they're going to be a lot because you have fat tails. And if you miss a few of these marginal gains, it's not a big deal. So you should run more experiments with a bigger significance level. Anyway, I wandered all over the place. I'm not sure I answered your question.

But no, it's a, it's. Look, it's a great digression and let me try to. I think we fully agree on this, but I'm trying to summarize some of this with some of what I believe, which is that like, I do think that focusing on P values and statistical significance is probably not the right thing to focus on. Instead, what I push businesses to focus on is what is the decision you need to make and then what is like the range of risk or reward associated with that. Like, what is the cost of making the wrong decision here? What is the potential benefit if it goes well? And try to put those numbers like as close to like dollar terms for the business or profit numbers as you possibly can. And that, you know, again, if you're like drawing the bell curves on the whiteboard with the product manager, like, that is never going to end well. If you can put it to be like, look, low case we lose $5, high case we, we gain $1,000,000, like seems like a good bet that is a lot easier for them to understand than like the dynamics of, you know, oh, what's the distribution? Or what's the false positive rate under the null? All of that stuff just ends up, I think, clouding the decision. And really as like data scientists or economists or whatever the title is, we should be helping businesses make good decisions. And, and so we should try to put these results into a framework that is like around the decision and around, you know, good case, bad case or like risk or, you know, what is the shape of returns that we might expect. I think that's a lot easier for business people to get their heads around.

Yeah, I think that's very well said. And I mean at the end of the day it's very easy for us as scientists. We get, everyone gets caught up in trying to measure lift. And that's not the real question. The real question we need to answer that the business cares about is what is the optimal level of marketing spend in general and then for specific channels. And even if we could get that to like 10%, we'd be super happy. Right? This is, it just needs to be better. And it's surprisingly hard to get to like, you know, that level of even being within 10% of the truth. And I think the other thing is being aware of where are the decision boundaries because like there's no point in having an argument about whether or not it's statistically significant if it's not going to change your decision. And so knowing where those points are and you know, sometimes I've wondered if we should just like T shirt size, the risk or the confidence on this. It's also if you live in a world where you have lots of RCTs, that's a much better world. And the more so randomized controlled trials. So the more randomized controlled trials or experiments or AB tests or whatever language you're using at your company, you have the better off you are. And for some things like brand marketing, they're surprisingly hard to run, they're surprisingly expensive and even major companies don't have a lot of them. You know, there's this whole myth that these big companies are running tens or hundreds of thousands of AB tests and they're probably talking about on site feature experiments, they're not talking about marketing experiments. Another thing we've done at Amazon that's really helped shift this conversation is we have a conversation about what we call value of experiments. And there's a paper by Hito Imbens and others based on their experience at Amazon that I think is publicly available. And so basically you can set up a dynamic programming game and compute the value of running one more experiment.

This is fascinating to me because I buy into this idea of the value of knowledge, but I feel like sometimes Again, communicating that to executives can be a challenge. And Amazon is very sophisticated, probably one of the most sophisticated organizations from the leadership down. But what would your advice be for other people, maybe working at other organizations? How do you have that conversation? What did you find? What are the explanations that resonate when it comes to helping people think through what is the value of additional experiments?

I think that's a hard question and I think that's something I certainly struggled with when I was at ebay and providing experimentation, thought leadership there. I think that we all struggle with it. It's one of these places where there's a great example of where as scientists it's very hard to put things into easily explainable language for leaders and executives and helping them understand that you're probably making the wrong choice today. And you can either continue to misallocate money or you can invest in an experiment. It's not a cost, it's an investment in knowing more so that you can make a better decision so that in the future you'll get more returns because you're at a more optimal spot. You're either spending less on marketing and not wasting money on marketing that doesn't work or you realize you're not marketing enough and you have to move up some response curve.

Great answer. I mean, I think it's easier said than done. So wish the best of luck to anyone listening to this who's going to go try it. But I think that's the best that you can do is hopefully, you know, repeat it a bunch of times and get the executives to buy in. Framing it as an investment I think is probably like the most concrete takeaway that I am going to take. I want to talk now a little bit more specifically about marketing. You've mentioned this idea of brand lift now a couple of times and brand lift, I think for me as a marketing scientist has always been a really frustrating topic because I think it's, you know, know, really, really difficult to measure what this long term effect of the value of a brand or the value of quote unquote brand advertising. I'm curious, like it seems like you've been thinking about this as well. So what can you share in terms of what, what is the right way to think about this? What's the right way to measure it? How do you, you know, how have you advised people to, to tackle this problem?

Yeah, again it's a super hard problem and a lot of the beliefs in the industry aren't based on causal studies. So they're anecdotal, they're like, hey, we went and collected data on a whole bunch of companies that just turned off marketing, brand marketing, and we compare them, but there's not even something like a matching estimator or any attempt to make this causal incredible. There were these split cable studies that were really popular in the 90s and 2000s, the Hugh Lodish literature that some of my friends would disparagingly refer to as being before the credibility revolution. And to make this causal incredible, there were these, you know, split cable studies that were really popular in the 90s and 2000s, the Hugh Lodish literature. And so people have a lot of beliefs in spider sense, you know, intuition about this. And we also. What makes this hard is we don't fully understand and we don't observe the cognitive pathways that go into building mental availability in your mind. And there's so many factors also that affect the value of that mental availability of brand equity, or whatever you call it, depending on the size of your company, the industry you're in, whether how mature your company's presence is in a market, you know, whether it's developed, transitional, or emerging. And another part of this that we know is that once you've built enough brand, following customer service is the thing that matters the most. And so you are a bolt on if you're measuring marketing to customer service. And so there's probably an interaction as well. There's an interaction with variable marketing, and there's so many things to unpack. And then most people have some notion of ad stock, so we think that this mental availability will decay over time or, or there's some carryover from period to period from what you've been exposed to in the past. And then people make functional form assumptions about response and saturation. So all these things make it super hard to measure. One of the best studies, in the opinion of the folks I work with, is the Shapiro, Hitch and Tuckman TV paper. And so I think that's like a very good reference. If you're like looking for a paper to get started, I'd start with that paper.

Can you summarize the method and the result there for the listeners?

Yeah, so they used a classic econometric approach with a ad stock model, but again, they weren't able to estimate the carryover effect in the ad stock. They just made a reasoned argument that it should be like 0.9. And then they ran a econometric model with T made a reasoned argument that it should be like 0.9. And then they ran a econometric model with TV data. And again, the other thing that's really hard too is variation in the data. And that's something we struggle with because, particularly for stuff like TV data. And again, the other thing that's really hard too is variation in the data. The industry is really not set up for us to measure it, and they probably don't want us to measure it. They want us to trust that it works. You know, Google and Meta, maybe they at least go through the theater of letting us think we're measuring it. But they also have challenges. Particularly there's a lot of literature now around randomization issues which we could digress into on another show. But so TV's super hard and that's something we struggle with because, you know, particularly for stuff like tv. And it may also not even be targeted at the demographics you care about, but yet we spend on it. And, and so we struggle with variation. Most of the TV variation is national. It makes it very hard to identify effects in these models. And it also means the industry is really not set up for us to measure it. And they probably don't want us to measure it. They want us to trust that you can only say something credibly down to a certain grain. But many of your, that it works, you know, Google and Meta, maybe they at least go through the theater of partners are going to care about stuff below that grain because they want to ask questions like, how do I optimize for this business unit in this channel in this country on the week of Thanksgiving or Christmas or Ramadan?

So we're in a world of uncertainty and there are a bunch of, you know, reasonable questions that marketers would like to know the answer to. And I would, you know, as a scientist, would love to know the answer.

To, but letting us think we're measuring it. But they also have challenges, particularly there's a lot of literature now around randomization.

That we practically can't answer, and most businesses won't even be able to approach answering. So, like, what do you.

Issues which we could digress into on another show, but so TV's super hard.

Do in that situation. How do you make decisions? How do you think about, like, well, is this a thing that we should be thinking about? Should we invest in TV or not? Like, what's your advice to these companies that are working in, you know, with a real lack of certainty around the importance or.

And it may also not even be targeted at the demographics you care about, but yet we spend on it. And, and so we struggle with variation.

Most of the TV return on investment of these. Of these different marketing approaches.

Yeah, that's a great question. And your chair is certainly easier to sit in than mine. So one of the things I did is I educated my executives to this like, lovely notion from the mostly harmless econometrics books of fuch and fact. So a question can be either a FUC or a fact. A fuch is a fundamentally unanswerable question and a fact variation is national. It makes it very hard to identify effects in these models. And it also means is a fundamentally answerable question. And so something could be a FUC for several reasons. One, you can't that you can only say something credibly down to a certain grain. But many of you get the data. And so even if you had ideal data, you couldn't answer this question. It could be a fook because it's unethical. Like, I have old nuclear weapons from the 60s. I'd like to know if they still work. That might be useful for deterrent. But I can't go set them off. That's bad. Partners are going to care about stuff below that grain because they want to ask questions like how do I optimize for this business? And so then the question is, how do I turn it into a fac with unit in this channel in this country on the week of Thanksgiving or Christmas, which is a fundamentally answerable question. And so I may be able to get extra data, I may be able to pivot to a different question that still allows the business to be unblocked. And so we try to triangulate and provide evidence in these cases or Ramadan.

So we're in a world of uncertainty and there are a bunch of, you.

Know, where it's hard to answer the question well. So for instance, like doing work around.

Reasonable questions that marketers would like to know the answer to. And I would, you know, as a scientist would love to know the answer to.

But you know, measuring the impact of brand. You may be lucky in your data where, you know, you change marketing policy markedly at some point and you can do an event study around that. Or there are other places where you can get lucky and find natural experiments in your data because that we practically.

Can'T answer and most businesses won't even be able to approach answering. So like, what do you do in that situation? How do you make decisions? How do you think about like, well, is this a thing that we should.

Of changes in policy at your organization or other things that let you find variation that's hopefully close to random? I'd love to say More. But I'm not sure when, like the legal people are going to get upset when they're here.

Totally reasonable. I mean, one of the things that I tell people is that, look, there's a lot of facts or Fuchs. Fuchs. There's a lot of Fuchs. There's a lot of fundamentally unanswerable questions. But as business people, we have to make decisions anyway. Be thinking about should we invest in TV or not? Like, what's your advice to these companies that are working in. And like, there's a lot of things, you know, about, like, I run marketing for my company and like, we place a lot of bets that are not based on data, they are based on strategy. And like, long term, you know, with a real lack of certainty around the importance or term, where do we want this company to go and what is the representation that we want to have in the market? And we're like, not going to be able to get a clean read on the ROI of this podcast, but yet we're here doing it anyway. And I think that there's a lot of context in which you have to sort of just make bets without data. You have to think about what is the return on investment of these, of these different marketing approaches.

Yeah, that's a great question. And your chair is certainly easier to sit in than mine. So one of the things I did.

Is I educate answerable and what isn't, and then, you know, make a decision anyway. And it's totally reasonable at some points to make decisions based on strategy. I think a lot of people have gotten caught up in this cult of data analytics and be like, we have to be data driven. And so we have to have data for every decision that we're going to make as a business.

Business.

And I just don't think that that's right again. You know, sometimes people think that it's crazy coming from me. I'm a data scientist. I've been working in data my whole career. But like, I truly believe that not every decision can be made with data. And really the best organizations are sort of wise and thoughtful in terms of thinking about, okay, what decisions need to.

Be made without dictated my executives to this, like, lovely notion from the mostly harmless econometrics books of F and fact. So a question can be either a fook or a fact. A fook is a fundamentally unanswerable question and fact.

And where does data actually make sense to apply to this particular decision?

Yeah, I think that's, that's really wise and it's important to include Appropriate disclaimer on the data science 10. Right. And the, you know, again, we're in this business of trying to place winning bets in this like casino in Monte Carlo and we're doing what we can to stack the deck in our faith is a fundamentally answerable question. And, and so something could be a fuc for several reasons. One, you can't and the last thing I'll say is there's a method for thinking about get the data. And so even if you had ideal data, you couldn't answer this question. It could be a fouque because it's on these huge investments under uncertainty that are particularly long term called scenario planning. And so that's something that's helpful. You can think about what are the likely cases of the world in a year or two and how the different decisions you could make might play out in those different scenarios. And that can help give you clarity. So companies like Shell Oil would use this to ethical like I have old nuclear weapons from the 60s, I'd like to know if they still work. That might try to understand, hey, should I open drilling in the North Sea? And is that a good thing to invest in? So you know, they're not going to ever have like some perfect data that tells them yeah, this is a good be useful for deterrent but I can't go set them off. That's bad idea.

I love that. And I think again it's reasonable.

And so then the question is how do I turn it into a fact?

What can we reason about? And the other thing that I recommend people do is can you rule out states of the world? Maybe you can't get a precise ROI on something, but maybe you can rule in like hey, this is good enough to keep doing or, or this is so bad that it's not worth doing any more of.

And maybe that is a fundamentally answerable question. And so I may be able to get extra data that's enough to like.

Make a decision and take the next step and you know, hope that you get more data later or have other opportunities open up to you.

Yeah, I like to triangulate. Right. If there isn't a lot of good data, you know, if I have a couple of these like anecdotal marketing puff pieces and I have some bad noisy RCTs and maybe I got lucky and got a natural experiment and like they're all largely in agreement, I feel better. If they're all pointing in different directions, then it's much more of a situation I may be able to pivot to a Different question that still allows the business to be unblocked. And so we try to triangulate and provide evidence in these cases for judgment. Early in my career I was working on this impossible forecasting problem and I went and talked to the resident wise statistician at Amazon, this is circa 2012. And he said, oh, you need a panel of experts. And I said, well what's a panel of experts? He said oh, that's the VP who signs the check. Important stakeholder where it's hard to answer the question well. So for instance, like doing work around, you know, measuring the impact of brand, you may be lucky in your data where. Right. And so one of the things I also really caution young data scientists is like if you're in these situations where you have to make these impossible decisions, you need to make sure that the leaders also have skin in the game so that you're not being used as a scapegoat if the decision doesn't pan out. You want to make sure you're protected against that.

Ben, I think super wise, you know.

You change marketing policy markedly at some point and you can do an event study around that.

I guess last question for you is what are the common mistakes or misconceptions that you see in marketing science that drive you crazy or that you think listeners should be aware of and should be thinking about as they're taking these techniques to their own organizations.

So I think the number one mistake, and this is really well said by a friend of mine, which is or there are other places where you can get lucky and find natural experiments in your data because you should spend to learn and you need to run RCTs to measure. And like really it's those two things of changes in policy at your organization or other things. It's like we I'm sure most of your listeners have the same problem I have, which is you deal with people who support RCTs in theory, but not when you want to do something really important to things that let you find variation that's hopefully close to random. Understand fundamental aspects of whether or not your marketing works and also if you get a result they don't like that they're willing to trust the science. Like imagine I ran an amazing RCT with like high power clean signal, got a great read and it showed that brand marketing did. Absolutely. I'd love to say more, but I'm not sure when nothing would I be able to turn off brand marketing? Probably not. Like the legal people are going to get upset.

Totally reasonable. I mean one of the things that.

I tell people and you know, how many of those experiments would I have to run? Or if I run a bunch of statistically experiments that are noisy but there's like, no, not statistically significant. How many of these do I have to run before someone will agree that, hey, there's no lift here.

Yep. I think your first point on that, about spin to learn, I think is actually probably the most important one. Well, is that, look, there's a lot of facts or Fuchs. Fundamental Fuchs. There's a lot of folks, there's a lot of fundamentally unanswerable questions. But as business people, we have to make decisions anyway. And again, for those listening, feel free to like, take the snippet of Ben saying that you have to do it because again, I think this is one of those things where leadership in a lot of businesses, like, they're down with running experiments, they like the idea theoretically, but when they start thinking about the cost, they're like, they sort of don't like that as much. But I think spending to learn, it's critical. And in the long term it can really help businesses in terms of optimizing. But it requires that upfront commitment to say, hey, look, and like, there's a lot of things, you know, about, like I run marketing for my company and like, we are willing to take some risk. We're willing to invest in and learning and so that we can use it to optimize the business overall.

Yeah. And since we were adjacent to marketing, we should say you should invest to learn.

Invest to learn. Exactly right. Much, much more compelling word, invest. Ben, this has been a wonderful conversation. Thank you so much for joining me. Really, really appreciate having you on. And yeah, good luck to all of our listeners who are going out and putting this into practice.

Yeah, thank you so much for the opportunity to speak with your audience. And I wish them all the best of success with their measurement. And I'm really grateful for your refreshing approach to.

We place a lot of bets that are not based on data. They are based on strategy and like.

Long term marketing measurement.

Thanks, Ben. Term. Where do we want this company to go and what is the representation that we want to have in the market? And we're like, not going to be able to get a clean read on the ROI of this podcast, but yet we're here doing it anyway. And I think that there's a lot of context in which you have to sort of just make bets without data. You have to think about what is answerable and what isn't and then you know, make a decision anyway. And it's totally reasonable at some points to make decisions based on strategy. I think a lot of people have gotten caught up in this cult of data analytics and be like, we have to be data driven and so we have to have data for every decision that we're going to make as a business. And I just don't think that that's right. Again, you know, sometimes people think that it's crazy coming from me. I'm a data scientist, I've been working in data my whole career. But like, I truly believe that not every decision can be made with data. And, and really the best organizations are sort of wise and thoughtful in terms of thinking about, okay, what decisions need to be made without data and where does data actually make sense to apply to this particular decision?

Yeah, I think that's really wise and it's important to include appropriate disclaimer on the data science tin. Right. And again, we're in this business of trying to place winning bets in this like casino in Monte Carlo and we're doing what we can to stack the deck in our favor. And the last thing I'll say is there's a method for thinking about these huge investments under uncertainty that are particularly long term called scenario planning. And so that's something that's helpful. You can think about what are the likely cases of the world in a year or two and how the different decisions you could make might play out in those different scenarios. And that can help give you clarity. So companies like Shell Oil would use this to try to understand, hey, should I open drilling in the North Sea? And is that a good thing to invest in? So you know, they're not going to ever have like some perfect data that tells them, yeah, this is a good idea.

I love that. And I think again, it's reasonable to think about, you know, what can we reason about?

And the other thing that I, that.

I, that I recommend people do is like, can you rule out, you know, states of the world? Maybe you can't get a precise ROI on something, but maybe you can rule in like, hey, this is good enough to keep doing, or this is so bad that it's not worth doing any more of and maybe that's enough to like make a decision and take the next step and, you know, hope that you get more data later or have other opportunities open up to you.

Yeah, I like to triangulate, right. If there isn't a lot of good data, you know, if I have a couple of these like anecdotal marketing puff pieces and I have some bad noisy RCTs and maybe I got lucky and got a natural experiment and like they're all largely in agreement, I feel better. If they're all pointing in different directions, then it's much more of a situation for judgment. Early in my career I was working on this impossible forecasting problem and I went and talked to the resident wise statistician at Amazon. This is circa 2012, and he said, oh, you need a panel of experts. And I said, well, what's a panel of experts? He said, oh, that's the VP who signs the check.

Important stakeholder, right?

And so one of the things I also really caution young data scientists is like if you're in these situations where you have to make these impossible decisions, you need to make sure that the leaders also have skin in the game so that you're not being used as a scapegoat if the decision doesn't pan out.