How to manage (and communicate) uncertainty in marketing measurement | Recast Office Hours #6

Foreign. So today we are going to be talking about uncertainty. Tom and I have prepared some lecture materials, really to sort of talk about some of the different interesting things, how we think about uncertainty, what, you know, we think is important about uncertainty, especially in the context of, of media measurement. This session is going to be slightly less technical, like we don't have interactive code or anything that we're going to be looking at. So mostly sort of talking at a high level. We've got, I don't know, probably about 20, 25 minutes of like lecture material prepared and then we'll open up for Q and A. I know that this is a topic that a lot of folks are interested in when it comes to thinking about uncertainty, dealing with it. So chat about whatever's most interesting to y'. All. Excuse me. So really four main topics here. The first is uncertainty and statistics. What does it mean? What are we doing? Second, uncertainty in marketing measurement. That's probably sort of focused a little bit more on exactly our day to day work. And then some thoughts on communicating uncertainty and talking about it with executives based on sort of our experience of doing this now for the majority of our careers. And of course, we'll have Q and A at the end. If you have questions as we're going through, drop them in the chat. So that way we'll see them and then one of the two of us will be keeping an eye on the chat and we'll surface them up for discussion as interesting questions come in. So first point is that inferential statistics is all about quantifying uncertainty in order to support good decision making. Basically all of the statistical methods that we know and love are pretty much all about how do we do this right? Like we've had the idea of an average or a mean for, you know, thousands of years. But a couple of hundred years ago, people started having these questions about like, okay, we take, take the average of something. How do we know if two averages are actually different from each other or if they're just different because of the sample that we happen to have drawn? And there's a whole bunch of work that has gone into this in industry. There's a lot of reasons to do it, but really the whole point of doing statistics at all is to try to quantify uncertainty. When we're doing inferential statistics, really what that means is quantifying uncertainty. That's what the whole game is about. But the problem as practitioners is that people hate uncertainty. Business people hate uncertainty. Human beings are not just like evolution. Evolutionarily speaking, we have not evolved to be able to work with uncertainty very well.

Right.

It is naturally against our nature to think in probabilistic or uncertain terms. Everyone naturally just wants to say, yes, it's going to happen or no, it doesn't. That is the easy mental shortcut to take when it comes to any problem. If we think about a lot of the work that like Nate Silver did with 538, he talked about this a lot where it was very frustrating to him that, you know, he would make these predictions or forecasts of what was going to happen and people would be like, oh, you know, Hillary Clinton has a 55% chance of winning. She's going to win. It's greater than 50. And so it means like, yes, but that's not at all what what he was trying to express or what the models were expressing, but our natural shortcut. And even myself as a trained statistician, I also fall into that trap. Like, I see a probabilistic forecast, I just like, look at it. Is it greater than 50% or not? Then it's going to happen or it's not. That's just the natural mental shortcut that everyone takes. And so really, one of the problems that we face as practitioners is how do we deal with that? And that's both sort of in statistical methods and then also in communication and decision making world. We need to be able to do both. So maybe to start off, I'll pose a question to the group. This is an example of the sort of thing that I've seen from channel managers and product managers. Hey, we just got back the results from our lift test. The lift test showed a significantly significant 3% lift. So we're going to double down our efforts in this channel. Did I put significantly significant? Oh, my gosh. Tom, you were supposed to edit this. Statistically significant 3% lift. Just fix that. Thank you, Jim. Okay, to the audience, what. What is wrong with this statement? Feel free to drop in, chat or come off mic. Don't make me call on people. I'll call on people. Okay. Jim says, no idea of the confidence interval. I think that that is definitely a problem. Antonio says, how much did it cost? That is another problem. Any others don't know if it's economically meaningful? Yep, exactly. It's sort of the cost question. What else you got, Corey? Diminishing returns. No mention of diminishing returns. Definitely a point. No details on the test. That's another really good one. Y' all bring up a bunch of points that I didn't even think about, but these are good. There were three major things that I sort of, three major things that I find problematic with that sort of statement, which again, I've seen many times in my work at Recast and even in prior careers, other work that I've done. So first is, is the economically meaningful, well, economic. Like, what are the economics of this, right, in the context of marketing? Well, so a lot of companies get started doing hypothesis testing in the context of AB testing for their product. And when you're AB testing for the product in general, you're looking at lift in some metric. And in that case, if you're comparing to business as usual, you do often only care about any positive lift. And so saying like, look, There was a 5% lift in conversion on this page when we did X, that often is enough to like make a decision. But in the context of marketing, it's not because we need to understand the ROI or the cpa. How much did it cost to get that lift? Just noting any positive lift at all actually isn't very helpful in the context of marketing because we always care about the ROI or the CPA in terms of understanding if that lift was profitable or not. So any lift is often just like not useful at all for making a decision. Two, because of that, right? Statistically significantly different from zero is generally not the appropriate statistical test. In the context of marketing, we very rarely care if something is just statistically significantly different from zero. If we are doing some sort of, you know, hypothesis testing framework in general, we want to make sure that we're comparing against, you know, the, the relevant decision making framework. Right. Is the ROI different from zero or is the lift above whatever the threshold would be for it to be profitable?

Right.

And so if you're thinking about doing like null hypothesis statistical testing in the context of marketing, we really need to make sure that we understand what we're comparing to and then that maps onto the business decision that we need to make. And then as Jim noted, there's no discussion of uncertainty. Really. These sorts of tests, we really, especially in the context of marketing, where almost everything that we're doing is pretty underpowered, we really need to understand the range of results, right. So that we can use that and feed that into our decision making system. So important thing to know about running experiments, right, is that they do not produce uncertainty.

Right.

When we're interpreting the results of a lift test in marketing or an AB test in product land or whatever, in general, we're not just getting a number, right? 3% lift, right? Is not what is actually coming out of the test. When we go back to basics and think about how these tests actually work in general, the information that we're getting out of the test is a range of values that are compatible with the data. And then we can apply a null hypothesis testing framework to that range of values and say, like, okay, how certain are we that the average is greater or less than some value? But really what's coming out of the test is a range of values that are compatible with the data. And so when we're interpreting those results, that's almost always the most important thing to look at. Again, especially in the context of marketing, where most of our experiments are fairly underpowered. And therefore these ranges can often be quite wide and can really cover a very large amount of space in terms of what's relevant for actually making decisions. So here I've plotted the results of two different tests, right, that you might observe in the context of marketing. So imagine some sort of incrementality test for two different marketing channels. This is the sort of thing that we often see, right? We have a test in channel A, we have a test in channel B. We have confidence range, confidence bounds associated with this test. Both of these test results, right, under a null hypothesis testing framework would say that they both show statistically significant lift. But the implication from these tests is not exactly especially clear as to what a marketer should do with it.

Right?

When I look at these results, I would look at channel B and say, yes, it did have statistically significant lift, but this channel isn't profitable.

Right?

The full range of results from this channel are below the break even roi. And so even though it had statistically significantly positive lift, it is not a good channel for this hypothetical company. And channel A, right, probably from some underpowered experiment, also has statistically significant lift. But the range of values that are compatible with the test results are enormous, right? It could be good, it could be bad. We just don't know coming out of this test. And so when I'm thinking about test results, this is often the problem that I'm thinking about and that I think marketers and people who work with marketers, statisticians, data scientists, data analysts, need to be thinking about when they're analyzing the results and then communicating those to the business. So it's sort of easy to start with experimentation. But now let's talk a little bit about media mix modeling and some of the different ways to think about uncertainty. So this is sort of our philosophy here at Recast, but it's something that I really believe which is that false precision is a, if not the cardinal sin in applied statistics. We think back to the first slide that we looked at, really like the whole field of statistics is about quantifying uncertainty. That's like everything that we're doing is about that. And so false precision is sort of just means that we've totally failed at our job.

Right?

The whole point of everything that we're doing is correctly quantifying uncertainty between the model and the data and what we need to do with it. And so this is a thing that we want to be very, very, very careful about. Because if you are falsely precise, you will often end up implicitly or explicitly making recommendations that will turn out to be wrong and will cause people to lose money or, you know, destroy business back. In the context of actually using statistics for decision making, what really matters is calibration, right? Which is to say, you know, do our results, can we actually use our results for good decision making? And when we make a prediction about what's going to happen, does it come true at the rate that we would expect in line with the uncertainty that we're reporting? So let's talk a little bit about calibration. What does that actually mean? So the idea of calibration has been around for a long time, but Nate Silver and538I think, did a really good job of popularizing this idea as being the right way to evaluate statistical models or being at least an important way to evaluate statistical models. And so this webpage still exists on 538 calibration. You can go check it out. It's got some nice little graphics here as you might be used to from that crew. This hasn't been updated in a very long time because I don't think 538 even exists anymore. But the webpage is still there. But anyways, the idea is that what you want to do is you want to one correctly express the uncertainty in some forecast and then compare those forecasts to actuals. And when you say that there is a 70% chance of something happening, you want to confirm that that happens about 70% of the time. And when you say that there's a 20% chance of something happening, you want that to happen about 20% of the time. And to the extent that you do that across all of those different probability levels, you would say that a forecast is well calibrated. And this is really, really useful for evaluating different forecasts and predictions. And then we'll talk in a second about how you might map that onto like inferential statistical models. Tom, I saw you come off mute. I'm not sure if you wanted to add anything.

No, I had to mute in multiple ways and so I switched to the physical flip mute. So apologies.

No, no, all good. So at recast, one of the ways that we think about, sort of the first way that we think about doing this is expressing uncertainty in forecasts, right? And so there's different ways of expressing this, but in general, you know, what we present when someone is using our tool to run a forecast is this widening cone of uncertainty out into the future, right? So actuals end on someday and then there's uncertainty about what's going to happen after that. And that widening cone of uncertainty reflects the true underlying uncertainty of the model. And basically what we want to see is that the actuals fall within the forecast 50% of the time within the 50% credible interval, and 90% of the time within the 90% credible interval, and 99% of the time within the 99% credible interval. That's the idea of having a well calibrated forecast. And this is the thing that we pay very, very close attention to internally at recast in the work that you're doing. This is a thing that we would recommend tracking religiously as much as possible. We're going to get into talking about, like how to talk about uncertainty with executives. But this idea of well, of calibration and consistently hitting the forecast and being able to make guarantees about, hey, you know, we consistently hit the 50% forecast 50% of the time and the 90% forecast 90% of the time going back into history. That's a really useful tool for being able to have these conversations with executives. And so we'll talk more about that. But this is a really, we think, important first step when it comes to getting your organization to think more holistically about uncertainty. So where does the forecast uncertainty actually come from? And where it comes from in the case of media mix model is from the uncertainty in the underlying parameters. So the way that media mix models work, right, is you have a bunch of parameters of interest that we're trying to estimate about how performant different marketing channels are and how long the time shift effects or ad stocks are, and, you know, the diminishing marginal returns curve, et cetera. And so the uncertainty in the forecast is really driven by the uncertainty in those parameters. So the more uncertain we are about how well a marketing channel performs, the more uncertainty will appear in this forecast, especially if we're planning to spend more money into the channels that are Most uncertain. And so basically, having the correct estimating the uncertainty for the individual parameters correctly is really important, both for being able to forecast well and then especially for being able to forecast well when we are going to intervene on the system. And when I say intervene on the system, what I mean is in the case where we are making a recommendation to change the budget in some way, and so good media mix models should be communicating that uncertainty to users. And why? Well, because, like, first of all, this is sort of the true way to communicate what's actually coming out of the model. But most importantly, because that's going to map directly onto the uncertainty that they're going to see when they come, when they're actually making a forecast. And so we. Communicating uncertainty is hard. We think about this a lot at Recast. I don't think we've fully cracked it in any, in any true sense, but we do try to show uncertainty where possible.

Right.

So you can estimate both the, you know, the ROI and the marginal roi and then also show the uncertainty bands that are associated with those estimates. And this can give users a helpful way to quickly see, okay, how relatively certain or uncertain is the model about this particular parameter. We also try to show how this changes over time. Again, because we believe that this is really important for actually understanding and making use of the model results. When it comes to calibrating a marketing mix model with lift tests, you really want to take that uncertainty into account. And so when you're thinking about the right way to do this, you want to be thinking about, okay, we have uncertainty from the lift test, we have uncertainty from the model. We want to make sure that those two pieces of information are playing together smartly. And I think it was our last office hours or one of the other office hours where we went really deep into how we think about doing that at Recast, when we think the right way to calibrate an MMM with a lift test is so definitely go and check that out. I think it's on YouTube if you want to go and review those results. So now I want to talk a little bit about how to communicate uncertainty and how we think about some of the right ways to communicate uncertainty, especially to, like, you know, business stakeholders and executives. Unfortunately, there's no easy answer. It is hard. And so if you have struggled with this in the past, you are not alone. There's no real easy way to do this again, because, as I mentioned, humans are just not generally naturally built to be able to handle uncertainty. And it's just not a thing that we like doing. And so you will, you will get pushback from executives, from business stakeholders, and so expect that, right? It doesn't mean you've done anything wrong. It's just very natural. There are four big tips that I want to talk through that in my experience, have really helped with being able to communicate uncertainty to executives. So the first, and we're going to, we're going to dive into each of these. But the first is to differentiate between what you believe or recommend and what the data say. Second is to be confident. The third is to map uncertainty onto actual business outcomes. And then the fourth is providing clear ways to reduce uncertainty. So let's talk about these one by one. So we often hear, you know, or I often hear from data science or statistics practitioners, you know, when I say, like, hey, look, you know, you should be talking about uncertainty. You should talk about credible intervals. And they say, no, no, no, no, no. My stakeholders, they don't, they don't want to hear about that. And if I talk about uncertainty, they're going to think that I don't know what I'm talking about, that I'm being wishy washy. And I think the easiest way to avoid that is to really separate between what you believe or recommend and what the data actually say. And I think that this is, if you confuse these two, then this is the place where you are going to come off as wishy washy. But when I am, someone's giving a presentation to me about something, right? I care a lot about what exactly do the data say, right? Do the data answer this question or not? How certain are we that this thing is true based on these data? And then separately from that, I also care about what that person thinks about what the data imply or what we should do about it. And I want to know both. And it's very important to me, actually that those things are separated. And I think executives do get this where you can say, hey, look, we ran this test, it did not provide us a lot of information. We're still very uncertain about what, you know, whatever. Like if this channel is really good or really bad, that's what the data say. And then you can pivot and say, that being said, I believe that strategically this marketing channel is really important for our business and so we should continue spending into it until we have more information. And that idea of both sharing what do the data actually say, uncertainty and all, and then pivoting to what do you recommend? What is your interpretation? What is the, like the business recommendation that's the right way to Separate those two things out and avoid that like, wishy washy problem or some of the feedback that you might have heard. When people talk about sharing uncertainty with.

Executives, I think that's an important point that I just want to double underscore, just uncertain. Given some properly calibrated uncertainty, there is a correct decision as long as you have a specific goal, right? So, for example, in, you know, optimal portfolio theory, given a basket of securities that have some risk, some covariance matrix, and you have some risk tolerance, there is a single optimal portfolio, right? Even though the returns of the securities are uncertain, there is a single best decision. Media mix planning is very similar to that, right? As long as your uncertainty is properly calibrated, the returns of your advertising investments will be uncertain. But there is a correct decision. And parsing through that uncertainty really is the job of recast our customers, the people that we work with. This really is. It is figuring out what the optimal decision is. And so I just wanted to underscore what Michael is saying here, which is that just because the data and the parameters are uncertain doesn't mean that the decision has to be. And this is an area where instead of sort of being on your back foot with respect to uncertainty, you really can be on the front foot. That's all I wanted to say. Thanks.

Excellent points. Point number two is to be confident, right? Again, be on the front foot. As Tom was just saying, uncertainty is a fact of life. Executives get that. You can use the stock market or the weather as examples. There are lots and lots of businesses that depend on the weather and depend on making bets about the stock market. None of them wait until things are certain in order to make decisions. Everyone understands that uncertainty is a fact of life. And so you should go in being confident talking about this. The more confident that you are, the more this is going to resonate. And again, executives get that uncertainty is a thing. They might not love it. Like, everyone would prefer to have a certain answer over uncertainty, but they understand that uncertainty is a thing. And so you should be confident when you're talking about it. The wrinkle here is that you do need to really understand what you're talking about, right? The place where I've seen people go wrong is they come in and they say, like, yes, you know, we got these results and this is the, you know, the P value. And the executive says, well, what does that P value mean? And then they say like, and they sort of panic because they don't actually really have a full grasp of what they're talking about. And so this is the place where I would say you do need to really make sure that you know exactly what the results that you are sharing actually mean and what their implications are. This is one of the reasons why I generally shy away from P values because I find them very non intuitive, very difficult to explain. If I'm in a meeting trying to explain a P value to an executive, I'm probably in trouble. I would much rather focus on a credible interval coming out of a Bayesian model, which I think are just very intuitive, very easy to understand, fairly easy to explain, and so a little bit easier to work with and to be confident when I'm having these conversations. So definite recommendation to both be confident, but then also really make sure you know what you're talking about so that if you do get questions, you can go that one or two levels deeper. Third recommendation, and this again goes back to what Tom was just describing about the optimal portfolio theory, is that we want to map uncertainty onto business outcomes and in particular do that under different scenarios. And again, this is a place where if you say like, oh, you know, we ran this test and there was a range of X to Y, you know, the executives might not love that, but they actually often do respond really well to like phrasing this in terms of scenarios. Seven times out of ten, this bet is going to be profitable for us. It's not a guarantee, but that's what we've calculated.

Right.

That sort of phrasing generally really resonates. I case in low case, right? In the high case scenario looks like X. In the low case scenario it looks like, yeah, down the middle, this is what we can expect. Those taking the results of some sort of fairly abstract or opaque statistical model and then converting that into here are what the outcomes that we, that we expect in the language the stakeholders understand, this generally tends to resonate and they really do get this. And so anytime that you can take some uncertainty and then map that onto a business outcome, and especially again, if you can do it the way that Tom was describing in terms of being able to map it onto some optimal decision under uncertainty, which you can in portfolio theory, and there's a number of other contexts where you can like really do that and actually make a true good recommendation, that's great. And the other thing to do is to try to put it into some sort of scenario analysis. And that's a thing that generally resonates very, very well with executive because again, they know that like, look, when we make a forecast and we say we're going to do $15 million in revenue next quarter. Like, they know that that isn't guaranteed. They understand that there's a high scenario and a low scenario, and they, you know, understand that there's different places that you might land. This is the sort of way of thinking about the world that they're very, very familiar with.

The question in the chat, well, not really a question, but from Tim that his stakeholders always ask, is it stat sig? So when you get that question, Michael, how do you respond?

I would try, Like, I would try to pivot the conversation. Honestly, I think most executives, what unfortunately, how I was taught and how most people of my generation and older are taught, and I would say probably even most people in general, is that statistical significance means that it's true. That is sort of what I learned when I was in college. That's what I think most executives think, where they think, like, okay, if it's not statistically significant, it's false, and if it is statistically significant, it's true. And so I would really try to pivot the executives away from just even asking that question at all. And so when I am asked that question, I generally say, that's not the right question to ask. The right question to ask is something like one of these business outcomes of, hey, how often is this going to be a good bet for us?

Right?

That, I think, is a much more useful framing of understanding uncertainty and statistics rather than, is it statistically significant? Because, again, statistically significant, like people say statistically significant, but they miss the rest of the phrase, which is often statistically significant different from zero. And in most business cases, zero is not actually the relative compare is not the right comparison. It's some other number. And so statistically significantly. Oh, my gosh, I cannot. I should have practiced this in the mirror. Statistically significantly different from zero is not the right comparison. And so it's just not even the right useful question to be asking. And so, again, I would really be trying to, like, wean executives off of thinking that way and pivot the conversation back to some business decision and the range of outcomes that we could expect as a business if we did X versus. Yeah, Tom, I don't know if you have a different answer. You've done. I know that you've done.

I mean, my feeling is, is that a lot of people, when they ask that question, don't really know the specifics of the question that they're asking. And if you did the, like, classic politician thing where you just answer a different question than the one they asked, like, if, if I, if somebody came to me and said, hey, is this, is this experiment stats egg? And I just said, you know, the results of the experiment are that increasing investment into Channel A will be profitable for us. Seven times out of ten. My sense is that that would probably answer the question, but I'm actually curious if others would agree with that because my feeling is that people, it's often people that aren't super deep in the weeds in stats land that are asking the sort of generic is it stats a question? They're not necessarily looking. That's just how they know how to phrase it. I don't think they're actually looking for a P value or a null hypothesis test. I think that they're looking for actually this level of detail and if you just go straight to it, it should be fine. But I'm actually curious what others think of that, if that's true or not.

I think it's a good question. While y' all are stewing on it, slash answers, Antonio, just as it's always statistics. Yes, it definitely is. While y' all are still. Got it. Put it in the chat. I think I've got one more point here that I wanted to hit on. Yes. So the last point that I wanted to hit on here is to provide clear ways to reduce uncertainty. And this could be, hey, look, you know, I know you executive are not thrilled about the amount of uncertainty that you're seeing here. There are things that we can do about it and providing that way forward can be very helpful. But it's important to also share that there's a cost to reducing the uncertainty. And so you also want to like, make sure to put it in context, we can run another lift test that is even higher powered. However, that will take n number of weeks, potentially as an opportunity cost of X or an explicit cost of Y. You just want to make sure that that's being shared. Another way to do this is to sort of phrase things in terms of like, look, we're going to make decision Y, which we think that seven times out of 10 is going to be a good bet for us. And then you can also say, look, there are, there's evidence that we will see as we are moving forward. If we're in the bad scenario or the good scenario, right by April, if we don't see at least X, we'll know that actually it probably wasn't that good. Phrasing things that way of even being like, look, we, we, we will have additional evidence as we move forward in time. There are other Sort of pseudo experiments or other types of evaluations that we can run in order to give us more confidence as we are taking these actions. That can be a really, really good way to help again get these executives comfortable with this idea and still be able to move forward.

Right.

Because that's often, I think, one of the big problems in a lot of businesses. They get this analysis paralysis where they're like, oh, there's uncertainty in this test and now we can't do anything. And again, that's almost never the case. Almost always there's a case where you can do something and you should do something and there are ways to get more additional information as you move forward in time and keep making decisions. And so making that clear can be very helpful to break the analysis paralysis. All right, I think we've got some more questions.

Just sort of a general anxiety or hand wringing about getting advertisers to think about profitability versus just stat seg. And the difficulty of getting to statistically significant at the level of profitability and not just at the level of zero. Right. If the null hypothesis is not is there any effect? But if the null hypothesis is like the ROI greater than our breakeven, even more difficult to get power at that threshold. So I think folks are sort of just a little morose about the possibilities there, which I think is fair. I think is fair.

I mean it's, it's definitely fair. But again, Tom, as you noted, the marketing mix problem is an optimal portfolio design problem. And so even if there is uncertainty, we can design optimal portfolios that still provide theoretically optimal results, again assuming that the uncertainty coming out of the model is, is, is correct. And so there are ways forward. And you can again take more of like a multi armed bandit approach to experimenting with different media mixes to continuously optimize even in the case of uncertainty. And so there are ways forward. But it is important, I think, to like, wean the executives off of the idea that like statsig means it's good and everything else means it's not. Because often we don't have that. Most of the time we are not going to have that. So you have to get them to understand that first and then you can start talking about, okay, how do we make decisions in the face of uncertainty? How do we place good bets? How do we evaluate whether those bets paid off or not? And the more that we can do that and the more that we can bring our executives along, I think the better off everyone is going to be.

The other option, of course, is just to adjust the parameters of static to the problem. Right? There's no reason the null hypothesis has to be zero. The null hypothesis could be your break even profitability. And there's no reason the, you know, p value has to be 0.05. It could be 0.4. Meaning that like if you feel like a channel has a 60% chance of being better than your threshold ROI, you could say it's statistically significant at those levels and just customize it for the specific decision making problem that you have. That's obviously a little bit challenging in terms of communication because people are probably not going to understand those parameters. But yeah, I'm curious if any folks have had some inventive ways to discuss or communicate uncertainty in the chat or in the group. Feel free to dump them in the chat or just take yourself off mute.

We were originally scheduled to only go to 140 my time. 240.

Are we at the time?

Oh wow, we're at time flew by. But I can hang out for another couple of minutes. If anyone has follow up questions or anything else you'll want to chat about.

I can as well. I'll be here. This is now recast office hours after dark, so feel free to take yourself off mute if you want to just chat.

All right, well we're not getting a lot of engagement here so we will wrap up. Thank you everyone for attending. Hopefully this was helpful. Really appreciate it. Good conversation and we'll see you at the next one. All right, bye.

Sam.