How to run good backtests & holdout forecast accuracy checks | Recast Office Hours #3

Foreign Good stuff. We're going to jump in, so welcome. I am Michael Kaminsky. With me is my co founder and your co host for these office hours, Tom Vladek. We are the co founders of Recast. We both have backgrounds in statistical inference econometrics, and we have a real passion for causal inference. And so if you've joined us for the for the past two versions of these office hours, you will have seen that we spend a lot of time talking about methods for doing causal inference. How do we think about variable selection, which I think was the first one, and how do we do parameter recovery analysis, which is the second one. And so the goal for all of these is to really share our knowledge and have a good discussion about your problems and your questions on the topics around causal inference. Hopefully be as helpful as we can for all of you practitioners who are spending a lot of time thinking about this stuff. So we've got I think over 70 people RSVP to this office hours. So in order to make this workable, a couple of ground rules. So the first is please mute yourself and stay on mute until or unless you're called on. Second is write questions that you have in the chat. Tom and I, along with Michael Cilimano and our team, Michael, we'll be monitoring the chat for questions. We will answer some things in the chat. We will surface some of those questions to talk about, live with other folks. And then just so you know, any questions that we don't cover during the office hours today, we will answer offline and share out those answers via email. So definitely drop those questions in the chat so that we have them. We can share those back out to everyone even if we don't have time to get to it today. So if you've been to these before, you know roughly how this is structured. Basically two parts to these to this office hours. For the first like 20ish minutes, Tom and I have prepared a little mini lesson on a topic that we care about today. We're going to be talking about holdout forecast accuracy checks. And then once we make it through that, we will open up the floor to Q and A and we will probably have a bunch of questions on holdout forecast accuracy checks. We can sort of dive into the weeds on anything y' all are interested in or just any topics around. Marketing, measurement, statistical analyses that you have. We're happy to just be as helpful as we can to y'. All. All right, so let's jump into the material here. Let's talk about holdout forecast accuracy checks or Sometimes we just call these back tests in the context of media mix models. This is a really, really important concept and it's a thing that we take really seriously at Recast and we spent a bunch of time thinking about the right way to do these types of checks. There's a couple of key points and I think we're going to hit on most of this throughout the lecture today. But I just want to highlight all of this up front. Point number one, in sample goodness of fit metrics are not useful for evaluating MMMs. They are sort of useful sometimes to look at like we look at in sample map or in sample R squared or whatever occasionally. But in general they should not be the first thing that you are looking at to evaluate the quality of your marketing mix model. You need to be looking at measures of goodness of fit from outside of the training data. So out of sample goodness of fit metrics. Point number two is that media mix modeling is a time series problem. So you can't use sort of some of the standard out of the box machine learning tools for doing cross validation that you might be familiar with from other contexts. Point number three is that information leakage is a huge problem. This is partly related to this being a time series problem, but also just in general because of the way that marketing works and because of problems with indigeneity and causal relationships within the system itself that we are trying to model. You have to be very careful about how you're handling these when you're doing out of sample forecast accuracy checks. Point number four is that as you're thinking about designing a backtesting regime, you should attempt to mimic the prospective forecasting problem as closely as possible. And then point five is you care about both accuracy and calibration and the calibration piece. The thing that I think a lot of people don't always spend as much time thinking about. We'll talk a little bit about how we think about it today and why that's important. All right, cool. So first point, why can't we use in sample goodness of fit metrics? And so in sample goodness of fit metrics, what are those? So it's like if you run a regression in R or Python or Stata or SAS or whatever, you will get output from that regression. You will have the coefficient estimates, you will have some P values and you'll have something like an R squared metric, right? And that R squared metric that you see coming out of the regression is an in sample goodness of fit metric for the data that the model has seen. And the Fit of the model. What is that? Goodness of fit. How well does that model explain the data? Right. You could also calculate in sample make mean average predicted error or I forget what the other ones are, MRSE or rmse, root mean squared error, whatever. There's a bunch of these different ones. When we talk about in sample, we mean calculating those metrics on data that the model has seen. The issue is that while that can tell you some, it can give you interesting insights about the model fit itself. It doesn't tell you anything about external valid validity of the model. Which is to say, how well does this model generalize to things outside of what the model has seen before? And when we're doing media mix modeling and we're doing causal inference, we, we care about external validity, we care about the model's ability to generalize outside of the data that the model has seen. Help us be able to predict the future, help us understand what's going to happen when we intervene, when we make changes to our marketing budget. And in sample measures of goodness of fit don't really help us answer that question. Additionally, media mixed models are very flexible models. They have often more parameters than data points. Because of that, we expect that in sample fit should always be quite good, right? It's pretty easy to generate really good in sample fit with most modern media mix modeling approaches. Because of that, if you're building your model and just looking at these in sample model fit metrics, you're often going to overfit to the data, right? We've seen this, you know, some of y' all have probably seen this before where an analyst is trying to build a media mix model and what they do is they just look at any point where the model doesn't fit very well and they had a dummy variable and say, oh, there was, you know, an avid here. And so we're going to use this dummy variable to explain away that miss that leads to overfitting. You can get very, very high R squared or in sample fit metrics, but you'll have very bad out of sample performance. The model won't generalize, you won't have picked up the true causal relationships in your data and you'll end up just misleading yourself. So what is a backtest or an out of sample forecast accuracy check? The idea is that you want to hold out some of your data from the model training process, train the model on all of the data that's not held out, and then ask the model to predict what is going to happen on the data that it hasn't Seen before. Right. So in this illustration that we have shown here for how this works for recast is you have some back test date, you chop off the last 30, 60 or 90 days of data that are available to you, you train the model up to 30, 60 or 90 days ago, and then you ask that model to forecast the data that it hasn' seen before. And then what you want to do is you want to compare the actuals what actually happened with what the forecast predicted. So people, especially people who come from like a machine learning type background, they often think, oh, you know, we can use cross validation to, to alleviate some of the problems of overfitting just the same way that I use for my other machine learning models. The problem is that because media mixed modeling is a time series problem, these sort of standard like leave one out, leave two out, whatever cross validation approaches don't work very well because in a time series problem, every day is similar to the days that are around it. Right. Today is like yesterday and today is like tomorrow. In general, that's going to be true across your whole data set. And so if you're just dropping out random days in your data set as the holdout, you're not going to get an accurate measure of how the model does it performing on new data that it actually hasn't seen before. Because just excluding Tuesday, but still seeing what happened Monday and Wednesday isn't actually a very good test because what happened on Monday and Wednesday is very informative about what actually happened on Tuesday as well. Broadly. Right. The idea is that those sorts of cross validation approaches, they don't mimic the use case of what we actually want to validate, which is prediction into the future. In the case of intervention, right. When we are using our media mix model, what we want to do is to be able to say, hey, if we make this change to our budget, here is what we expect is going to happen to our conversions or our revenue. That is the actual business use case for doing this whole project. And so we want our model evaluation metrics to mimic that problem that we are facing as a business or an analyst. Right. What are the actual decisions that we expect to make as a business and how can we set up our model evaluation metrics to mimic that exact problem? Tom, anything to add on any of this so far?

Yeah. So typically, I mean, just consider the problem about leave one out cross validation and you have data through August 15th. Well, what leave one out cross validation would do is it would be just as likely to leave out August 15, which would actually be a prediction task where you have data through the 14th and then you're predicting the 15th, but it also would be just as likely to omit August 14. And since you know what August 13 is and you know what August 15 is, interpolating what August 14 would have been is actually quite an easy thing, right? And so you can't use simple leave one out cross validation because observations are not independent of each other. And if you have data that's sort of in the middle, that's not really a prediction task, it's sort of a filling in the blank task because you have observations on either side of it. So in order to do cross validation in the time series context, you need to leave the future out. You can only have data before a certain date, but not data after that time period, which makes it substantially harder. The other issue just with time series in general, is that since data is not independent across time steps, you actually have substantially less data than you think. Normally, if you're looking at say a set of a Data set of 800 customers, each customer is sort of a totally new person, right? Whereas if you have a data set of something like 800 days, around two years of data, each day is not a totally new day. So there's a question of in which case do you have more data? You have substantially more data if you have 800 customers than you do if you have 800 days, because each day is going to be very similar to the day before it. And that exacerbates the modeling challenge because you have typically a very big model and you have very little data. And if you have a very big model and very little data, your in sample fit is typically always going to be very good, but often at the expense of out of sample fit. And that's why you really need to know if you're out of sample fit is good, because otherwise you won't be able to make predictions and optimizations.

Thank you, Thomas. So, moving on, let's talk a little bit about information leakage. Information leakage is always a problem when you're doing machine learning Cross validation problems like thinking about, oh man, is there. Information leakage here is really important. It's especially important in the context of doing media mix models and trying to run good back tests or good holdout forecast accuracy checks. So broadly, information leakage happens when data that you won't have, when you're actually doing the real prospective forecasting exercise, leaks into the model, into the, into the training data. So you're not really getting a good sense for how well is this model going to perform because the model is cheating when you're doing this exercise. This happens a lot. Like hedge funds spend a lot of time thinking about this when they're trying to back test trading algorithms, right? Which is like, oh, is there information leakage? Is there stuff that we know today in our data set that is causing our model to overperform when we're back testing it, but where we won't get this performance in the future? This is a problem because it means your evaluation metrics won't be correct, right? You will be overstating the accuracy that you can actually expect to get when you're using this model in production. And it's just a really hard problem. It's very tricky to reason about. A classic example from like hedge funds in trading is you might train an algorithm using, you know, your data set of the S&P 500 stock performance for the last 20, 20 years. But you have to be very careful about this because which stocks are included in the S&P 500 changes over time. And so if you just like looked at the companies that were in the S&P 500 today and then looked at their stock price going back the last 15 or 20 years, you would get this totally wrong. Another thing is that some companies go out of business and so not only does the, the constitution of the S, the S&P 500 change, but like companies go in and out of business. And this has burned hedge funds before where they train the algorithm, but they excluded all of the companies that had gone bankrupt or had gone out of business. And they ended up like, you know, really hosing themselves and losing a bunch of money because their back tests weren't accurate, because it wasn't including that like those companies that dropped out of the sample or that should have dropped out of the sample, but the hedge fund never included them. So these can be very tricky. We're going to talk a little bit about ways that this can be very tricky in the context of media mix modeling and marketing analysis as well. So listed here a number of variables that can leak information that you need to be careful about. First one, seasonality and trend estimates. This one actually maybe is like a little bit non obvious as to how this might happen. It turns out that one of the popular open source packages for doing media mix modeling, Robin, actually doesn't handle this correctly. And so we're going to talk about that in just a second. But it's a good example of the way that information leakage can happen and you aren't getting true Holdout checks the way that you might think that you are. Variable number two, branded search activity. If you tell me for a brand how much, especially for like an E commerce brand, their branded search search activity over time, I can give you a really good estimate of what their sales are. And that's because branded search activity is a very, very bottom of funnel metric. And so both branded search activity or something like website traffic or foot traffic contains a lot of information about that business performance in it. And in fact, the problem with those variables, both number two and three here is that they are driven by other marketing activity. So if you tell me how much site traffic you had, it's sort of like if you tell me, hey, we know how many add to carts we had on our website. Well, once I have that information, it's really easy to predict revenue. But when I'm doing a forecast today about our business performance next quarter, I'm not going to have that information. Right. Branded search activity, website or foot traffic are all things that are actually driven by our marketing activity and they're very down funnel metrics that we won't have when we're trying to do a forecasting exercise today. Other similar variables are things like affiliate and coupon code use.

Right?

Those things happen after a purchase. And so again, if we know about the affiliate code use, it's pretty easy to predict revenue because those things are very tightly related. We only pay out affiliates after someone makes a purchase. But that means that again, when we're doing a forecasting exercise today about what's going to happen next quarter, we don't know how much affiliate or coupon code use, we're going to have refer a friend activity exactly the same. There are a bunch of other examples like this. Practically what this means is that if you use these variables when you are trying to do a holdout forecast accuracy check, your model is going to be cheating. It is going to be using information that it actually shouldn't have and you're making the problem too easy on the model. Tom, anything to add to this?

No. I mean the affiliate is like probably the most egregious example because in this particular case sales do not or spend doesn't cause sales, but rather it's the opposite, sales cause spend. Right. So if you are using that as an explanatory variable to predict sales, you have gotten the model totally backwards unless you're doing some accounting for it. But you cannot assume to know over the next couple of months what your affiliate spend is going to be because affiliate spend is going to be an outcome of your other activity in your marketing channel. And knowing and assuming to know about how your affiliate spend will perform is assuming to know a lot about the future. And so if you're putting that into the model, especially if you're back testing, you are going to have a lot of excess precision in the model that's not going to be there when you are actually forecasting. The same is true for everything else that's been put in here. You don't control directly how much branded search you will spend. That is an outcome of people going onto Google and searching for your branded keywords and that contains information about how effective your marketing program has been. And so if you assume that you will know in advance how many people are going to search for your branded keywords, you are also assuming you can know in advance a lot about how your marketing is going to perform, which is the very thing you are trying to predict. So these are subtle problems, but in practice it makes the prediction problem really challenging. And people often just throw everything that they have into a model and use that for back testing, which is not appropriate.

So let's talk about how this shows up in Robin, right, because this is a really interesting example. So at Recast, we often compare recast predictions with other open source packages. One of them is Robin. And when we cracked open the hood on Robin to do this, we sort of realized that the, the, the built in tools for, for doing out of sample forecast accuracy actually have a real information leakage problem in them that you need to be very careful about if you're using this tool. So this is an illustration actually that one of our research statisticians put together showing what this problem is. So I just copied it over here into this presentation because I thought that it was great. Basically the problem is that Robin has sort of two different stages of model fitting. In stage one, it estimates seasonality and trend components using an underlay another underlying time series forecasting package called Profit. And the problem is that no matter what you're doing in Robin, the Profit always sees the full data set. And so no matter what you're doing, Profit always trains that first. When it is doing holdout forecast accuracy checks, it first shows the full data set to Profit, takes the output of Profit, plugs that into the rest of the MMM fitting routine and then only cuts the data off for the holdout from the MMM fitting routine. And so what this means is that there's information leakage happening here where Profit, the one stage of the MMM routine and Robin is seeing the Full data set, but it's telling you that it's held out. And so there's an information leakage, leakage problem happening here. And so at Recast, when we went to go and actually do real holdout training, what we found was that when you actually exclude that data from the profit stage, the holdout accuracy gets a lot worse from Robin. And that's actually what you can expect to get when you're doing a prospective forecast over the next three months, over the next six months. And so you have to be very, very careful about when you're looking at these results. And even when you're using some of these open source packages.

There's two questions from Calvin and from Jeff that have to do with information leakage. I just want to just quickly get to before we move on. So Calvin, your question on Branded search, what about in certain categories with highly competitive branded terms where spending and branded can drive conversions? Just simply being pregnant present. And then Jeff, you asked about using questions like inflation. So both of these questions, although they're seemingly unrelated, have to do with the question of exogenous factors that impact the model and endogenous factors that impact the model. So the goal of an MMM is to predict the performance of your marketing activity within the world. Right. And so factors like inflation will impact your performance, but we're not really attempting to predict inflation. Right. And so the problem with using inflation in your MMM is that if you want to make a forecast, you have to either predict inflation or assume inflation. And I think predicting inflation is well outside the scope of what an EMMM provider should ever claim to do. But so you would be in the world where you would have to assume inflation. And I think that that's fine, but it's just something, it's a trade off that you would have to make. So in some cases you might want to have in your MMM some exogenous factor like inflation or like interest rates that you include. But what you have to do is you have to sort of make a number of scenarios and say, well, if interest rates are high, this is the forecast. If interest rates are low, this is the forecast. But you can't really presume to know what interest rates are going to be ahead of time. You'll have to run a number of sensitivities. And so where this links with Calvin's question is the extent to which you should treat Branded search as something that the model predicts or as something that you treat like this inflation exogenous factor will depend on, on the extent to which branded Search spend in that category is driven by your marketing activity. Like, can you move the needle, or is it just something that happens to you every month? Right. And it's driven really by the category. To the extent that it is just driven by the category and things that are outside of your control, then I would say you should treat branded search like one of these exogenous variables like inflation, like interest rates, and either don't use them at all in the model, or use them and then run sensitivities when you run forecasts, but never presume to know what they're going to be. If you can really move the needle with your branded search spend, then it is something that the model really should be predicting because it is endogenous to your marketing activity. And so that's how I would delineate the answer to that question.

I'm answering a question in the chat right now. Thank you, Tom. I think that was a great explanation. Okay, want to talk about some of these last points here before we sort of really open up, open up the floor to Q and A. So this is the thing that I don't think everyone necessarily agrees with, but we believe quite strongly here at Recast, which is that good forecasts should come with uncertainty.

It's.

It's really hard to evaluate a forecast if there's no uncertainty associated with it. And I think this feels very natural to most people when they are looking at something like the weather, right? Where it's like, there's a 40% chance of rain tomorrow. And while that can be frustrating to you, if you're trying to plan the beach trip, you're like, what does that mean? Is it going to rain or is it not? The nice thing about it is that it is in general going to be more accurate than just saying, yes, it will rain, or no, it won't. And over the long term, you can evaluate whether or not the model is well calibrated. Does it rain 40% of the time when we say it's going to rain tomorrow? Yes or no? That's a very testable, testable assertion that you can make. And so that allows you to really score your accuracy over time, which we think is really important to doing these sorts of forecasting exercises in the real world. So I've put up the example from, again from weather that a lot of y' all will be familiar with, where you see the path of the hurricane. And unfortunately, this is funny because, like, a lot of people see this chart and they actually totally misread it, and they think that what this is showing is that the hurricane is going to grow in size like 100x over the course of 120 hours, which is not what this is showing. What this is showing is saying, hey, based on our models, here is where we expect the hurricane will. Where the center of the hurricane could be over the next 120 hours. Right? And so we see this sort of widening cone of uncertainty as time moves forward, because we are more certain about. We are very certain about where the hurricane is right now. That means we can be pretty certain about where it will be, you know, 30 minutes from now, but we're a lot less certain about where it will be six days from now. And so this widening uncertainty is what you expect to see with forecasts. And it's. We think that getting to correct calibration, which we're about to talk about, is actually really important. So where I first started really, like learning about calibration was actually from reading a lot of Nate Silver's work from 538. So for those of you who are maybe like, very young and aren't familiar with 538, which has now been sort of sold off and sold around.538 for a very long time, was sort of the premier prediction website. They did a lot of work on predicting the outcome of sporting events, but probably got most famous for their work in predicting elections and specifically, like, political outcomes. And what538 made a big deal about was one, predicting uncertainty, right? There's a range of possible outcomes that the model is compatible with, and here's what those are. And then two, scoring themselves on calibration. And so when we say calibration, what we mean is that the model's uncertainty is an accurate reflection of what happens. And so here I have moused over this chart from FiveThirtyEight where they say, okay, in this bin, these were predictions that we made that we said had a 60% chance of happening. And on average, they actually happen 67% of the time. So that's not perfect calibration, but it's pretty good. And so in general, when people are making assertions about the future, you want to understand how well calibrated they are, right? If we present a 90 confidence range in some forecast, do the actuals fall within that range 90% of the time? That's what you want to see. If there's a 50% range of the forecast, do the actuals fall within that range 50% of the time? That's what you want to see. I often have, like, sales calls with people who are looking to get into marketing mix modeling. And they're like, well, how accurate is the model? And I'm like, I don't know. How stable is your business? Right. Really stable businesses, it's pretty easy to get a very like, quote unquote accurate model with very little amount of error just because the business is very stable over time. Right. We work with these businesses where, you know, most of the most of their sales come very consistently over time just because their primary distribution channel is Walmart and they just get a lot of sales from being on the shelf at Walmart, it's really easy to get a very accurate forecast. We work with other businesses that are highly impacted by like random events that are totally outside of anyone's control and totally impossible to predict. And there's just going to be a lot more error in those forecasts no matter what you do. They're just very different businesses. And the amount of like total random variation in those systems is different. At Recast, what we care about is that our forecasts accurately reflect the true amount of uncertainty in the system. If we give you a fairly narrow 50% confidence interval for the forecast, the actuals fall within that 50% of the time. We give a fairly wide interval, the actuals fall within that 50% of the time. Whatever those numbers are, that's the thing that we really care about and that's really how you should be judging forecast accuracy over time.

Basically. If you go back one slide, I'm going to introduce crps here. A big problem with accuracy measures when you include uncertainty is that most of the accuracy measures that we all use, like root mean squared error, mean absolute error, mean average predicted error, assume that for every observation there is a single prediction, right? There is a good question of how much error is there. If the actual was 0 and the prediction was anywhere between negative 1 and 1, right? Or between negative 0.5 and 5 or some normal distribution about 0, there's a really good question, like, what is the error error? Was it negative one? Was it one? Was it two? Because that's the range of the forecast. Is it one? Because that's the average of one in one. So when you have multiple predictions or a range of predictions for every single observation, which is what we have, then you need to look to other metrics for your accuracy measures because some of them will break down. Now I'm going to actually share my screen, Michael, if that's okay, and just walk through a little bit of code. So if you're not familiar with RStudio don't worry too much. But let's say our actual result is zero. And we have model one, which had predictions of one and one. So we had some model and we had some base case and some other case, and the base case was that you'd get one, and the other case was also that you would get one. I know this would never happen, but just bear with me. And then model two had predictions where the one possible outcome was negative one and another possible outcome was one. Okay, so I have two models, model one and model two. One predicted one and one and another predicted negative one and one. I think most people would agree that in a Sense Model 2 did a slightly better job of predicting zero because it has it bracketed. Now, if I look at two standard error metrics, though, for evaluating Model 1 and Model 2, what I will find is that the mean squared error of Model 1 is going to be 1. Right, because both are off by 1. And the mean absolute error is also going to be one. And so my two measures of accuracy for model one give me one. And the CRPS score, cumulative rank probability score, which I'll get into in a second, will also give me a score of 1. So it has an average error of 1 according to all of our metrics. Now let's look at Model 2 similarly. 1. This is getting a little boring. Mean absolute Error is also one for Model 2. Now, what happens when we do CRPs? Sample 0.5. So it matches our intuitions that this model here, which is in a sense bracketed the actual results, even though each individual prediction is just as off, is slightly better to use than this model, which is biased. All of the predictions are to the right of the actual result. And so this is why we use this measure. And this measure is a little bit hard to define. I'm not going to get too deep into the weeds on what it actually means, but what it does is it compares the probability, the distribution of predictions that you made, with actually the cumulative distribution function of the predictions that you made, and it compares it with the actual result in such a way that when you have a score of 0, your forecast is perfectly accurate. And it gives credits to reductions in both bias and variance, which most normal prediction measures or accuracy measures won't do in a very coherent way. Right. According to the normal error metrics that we had, they couldn't disambiguate between a model that was always off to the right and a model that had the actual squarely bracketed. But CRPs, the cumulative rank probability score can do this. And it's actually no surprise that this measure actually comes mainly from weather forecasting, where big models that produce lots of simulations are the norm and uncertainty is really baked into the predictions that they make. But, and this is what we use as well because we're in a very similar position. The upshot here though is really that good predictions have uncertainty, but predictions with uncertainty need a better error metric. They need error metrics that take into account both the bias and the variance of the prediction and not just assume it's a point estimate. CRPs cumulative rank probability score is one option. It's the option that we use. We think it has a number of desirable properties and we'd encourage you to sort of check it out.

All right, well, we have seven minutes remaining, Tom, if you want to just go ahead and stop sharing your screen. What questions do you all have? So Juan Carlos asked a great question. Apart from validating via forecasting out of sample, do you validate the coefficients of the model itself? Geolift tests specific channels. Answer is yes. Geolift testing is a great way to validate the model and especially to calibrate the model results to make sure that it's anchored to something in reality. There are a bunch of like edge cases here and ways that this can go wrong. You need to be quite careful when you're doing this. But broadly the answer is yes. So we think about checking for internal validity with parameter recovery, which we talked about last week and if you didn't see last week's, if you weren't or not last week, last month, I don't know how often we do these. The last one of these that we did, we talked about parameter recovery exercise. You can find the video on YouTube if you want to take a look at that. That helps to check for internal validity, for external validity, validating with experiments and whether that's sort of treatment control, structured experiments like a geolift or even less formal, less formal experiments, just like a Godard test in a channel or increasing spend into a channel next month, those all count as experimental manipulations. Those are all really good ways to validate the model externally as well, and we definitely recommend them. And the uncertainty that coefficients are also using the CRPs. So that is a totally reasonable way to calculate the overlap. Well, I guess in general, the thing to think about, I'm going to restate this. The thing to think about with experiments is that experiments also don't necessarily tell you the truth. Right. They don't give you the point estimate of this is the actual value. What they give you is also a range of uncertainty. And so when you're doing a geolift experiment, what you're going to get back from the geolift experiment is not, hey, the true CPA of this channel is $75. What you're going to get back is something that's going to say, hey, there's a 90% confidence range for this test where the CPA that is consistent with the Data ranges between $25 and $175. And so you wouldn't necessarily use the CRPs to score the accuracy. What you want to do is you want to look at the overlap of the uncertainty ranges between those different methods. Tom, did I get that right? I was sort of riffing there. I just want to make sure that you agree with me.

Yeah, you did, you got that right.

Anuj has the question if two or more models need the same level of accuracy but give you completely different row as for a channel, how do you know which one is right? Great question. You don't, you cannot, you cannot determine that. At least with a one shot analysis here, those two models are going to be sort of equivalent from this scoring framework. What we would recommend doing would be to try to take some action, whether that's by running a geolift experiment, sort of like what Juan Carlos was sharing, or by changing marketing spins, come up with some way of generating a falsifiable hypothesis and then running that experiment to try to see, okay, which of these two models is actually more consistent with those results that we're seeing. So you have model A that says branded search is your most effective channel and model B says branded search is terrible. Well, let's go run an experiment right next month let's increase our bids for branded search and see what happens. And hopefully that additional variation will allow us to rule out one of those two different media mix models. Doing that in a structured way I think is the best way to break the deadlock, actually using the MMM instead of just trying to like only use backward looking data analysis techniques, actually using the mmm, running experiments, running tests, trying things out is the best way to really try to validate the model's results. So Jeff is asking a question about decomp RSSD and how that's used in Robin. So for those who don't know, we've written a good blog, we've written a couple of good blog posts on decomp RSSD and exactly what it's doing. The people who wrote Robin would not describe it this way, but I'll describe it in my language and I think it's fair. Basically, you could achieve the same thing as Decomp RSSD by putting fairly strong priors on the model and a Bayesian framework that the current budget is already optimal. Effectively, what Robin is doing is it's starting from the point of saying we're going to assume that the current budget allocation is optimal and then we're only going to deviate from that where we have strong signal. And so you can achieve that same assumption or set of assumptions by using priors in a Bayesian framework. Robin just happens to do it in this sort of. What I think is sort of an opaque way of making the same thing happen. So you could achieve the same thing. I don't actually think that that's necessarily a good assumption. Maybe in some cases it is, or some analyst or some brand wants to make that assumption. But if you wanted to replicate that in a Bayesian framework like Recast uses, you could structure the priors that way. But that's effectively what Decompressd is doing. All right. And Jim says that that's also what lightweight MMM does. So that would make perfect sense. If you're using these frameworks, make sure you understand the assumptions that are going in. This stuff is important to get right because it will definitely shape the results. So make sure that you understand what are the default priors going in or what is the optimal. The optimization algorithm that is being used. Those end up boiling down to effectively the same thing. Very, very important stuff. We are out of time. Thank you so much for the great participation. It looks like we just got one more question in that we are not going to have time for. We will save it. That will get shared out via email. And yeah, really appreciate the time. Really appreciate the engagement. Hope you all have a great rest of your week. What are we? Thursday and a great weekend. All right, bye.