What is parameter recovery and why is it important | Recast Office Hours #2

Foreign.

Okay, so want to keep this efficient as always. We've got, I believe, 45 minutes here and a lot of stuff to get through. And then I know that y' all are going to have a bunch of questions. So as we jump in, want to quickly introduce ourselves. So with me is my co founder, Tom Vladek. And I am Michael Kaminski. We started this company recast. We both have backgrounds in statistical inference and econometrics and a real passion for causal inference. This is a thing, causal inferences, where we have both spent basically all of our careers. The idea is how do we use data and experiments to learn true things about the world and what are the methods for doing that? And this is a field that has developed a lot over the last 10, 15, 20 years. Back when I was first learning statistics, we honestly used a lot of really bad methods, things that just honestly don't work. And when you look back at a lot of the research that was done 10, 15, 20 years ago, you find that there's a lot of stuff that people thought that they learned. Articles were published, headlines were written, entire books were published that all just turned out to be totally false. And so Tom and I care a lot about thinking about what are the methods that prevent that? And specifically what are the methods that prevent that in the domain of marketing science, which is where we're specialized right now. So anyways, really excited to have you all and to share and talk about some of these topics. So for office hours, we've got a bunch of people already on the call. A bunch of people are probably going to be joining over the next couple of minutes, so just want to remind you all the right way to engage with such a large group. So please mute yourself and stay on mute unless you're called on at some point. You can use the hand raise feature if you want to ask a question or, or drop questions into the chat, which is actually probably the best place. Questions that are dropped in the chat, we can respond asynchronously and we'll be able to save. And we will send out an email to the group with any questions that do get answered. So that's the first place to go. Anyways, that's it. We're really excited about it. So we'll do the first probably 20 or 30 minutes with prepared content that Tom and I have been working on. And then after that we'll open up for sort of true open Q and a great cool Parameter recovery analysis is the topic for today. This, as I mentioned, is a thing that I, you know, didn't really under. Like, this was not taught to me in school, this idea of doing parameter recovery analysis. I sort of stumbled on doing parameter recovery analysis at my first job where I was working as an, as an analyst at this company called Analysis Group. And I started wanting to like really pressure test the different types of analyses and models that we were working on and make sure that I really understood what was going on. And the way that I did that was by simulating data and then simulating data and then making sure that the analysis got to the right answer. And what I found a whole bunch of times was that it actually didn't get to the right answer. Like nine times out of ten it was like it didn't give me the right answer that I was expecting. And so then I had to go in and really dig in and figure out what was going wrong. Was it the code, Was it my assumptions, what was happening? And that was a really, probably the most formative process in my entire career in terms of actually learning statistics and how these methods work. Right. Instead of just calling some statistical algorithm. At that time I was writing a lot of Stata and SAS code, so some built in function in Stata and sas, making sure that I really understood what was happening under the hood. And today we're going to talk about how we do this process in general and how you can use it in your work. So key points, key takeaways from today.1. Every statistical analysis that you're doing should start with parameter recovery. This is truly the most important part in the process of doing a statistical analysis. If I don't do this, I never feel like I'm actually standing on solid ground when I'm presenting the results of an analysis. It just absolutely should be done. We talk a lot about doing this in the context of media mix modeling, because that's what we do here at Recast. But really any statistical application, this is a thing that you should be thinking about doing. Framer recovery helps to identify and isolate bugs, which is great. But even more importantly, it helps to identify core modeling. Core subtle modeling issues like unidentified unidentifiability is one of those things which is like, it sounds theoretical when you just talk about it. And it's hard to know if whatever you're doing has this problem of unidentifiability. But parameter recovery will make this very, very clear immediately that you are facing this problem. And then the last is that once you get in the habit of simulating data, it's just a really powerful tool to have in your toolbox of this idea of we're going to simulate data and then we're going to run an analysis. And then once we get to having actual real data, the work is very trivial. It's a really just good tool to have in your toolbox. And this is often the first step of many people in their journey towards using more formal Bayesian statistics. A lot of the framework of Bayesian statistics is built around this idea of a data generating process, which is to say we are generating some data. Doing synthetic data generation via simulation is often the first step towards starting to really take this theoretical approach to the world. So all good things for you all to be thinking about. Want to sort of call out why this is important because we see people doing this the wrong way a lot. And this is a thing that I see on LinkedIn all of the time. People are like, I came up with some new method that solves some really hard problem and I tested it on real data and the results look reasonable. And so I know that it works. And that is just like not a valid way to develop a new statistical methodology. And it's like not a useful claim at all. Like, are the results reasonable Is not helpful. What we always have to do when we're evaluating some new method is make sure that we can do parameter recovery. When we know the true results and they look like X, can this new method actually recover those results? When the results look different, can the method recover those different results? That's the thing that actually tells us like, hey, when we're using this in the real world where the results truly aren't known, can we get to the right answer? So when do we actually do this? Important to know. We do this when we care about statistical inference, right? And this is a, you know, I talked about causal inference at the beginning. This is what we're doing in the context of media mix modeling in general. When we say that we're doing statistical inference, what we mean is that there's some unknown parameters about the world. What is the true effectiveness of Facebook prospecting ad dollars? And, you know, since we don't know what the true answer is, whenever we run any statistical model on data, it will spit out a number, but we don't know if that number is right or not. So we're doing this in the context of statistical inference when the true value of interest is unknown, it's not as important or relevant for like prediction only type problems. Right? If you're in the machine learning world and you're only doing prediction, this stuff matters a lot. Less the individual parameters don't matter unless you're doing again, some sort of like explainability work, at which point you're doing inference again. But in general, we do parameter recovery when we're working with statistical methods and we want to understand their properties. When do they work well? When do they not work well? There are trade offs in the world of doing in this world, no single statistical method works for every single different problem. And so a really good thing to use these techniques for is to understand where does it work well and where does it not. Right. This model will work and give us results when these things are true. If these other things happen to be true, we're going to have problems. That's a totally valid thing to say. And honestly, I think the best thing to say at the beginning of doing some analysis are actually using modeled results. As long as these things are true, we will get valid results.

Cool.

So we've been talking a lot about parameter recovery, sort of theoretically, why do we do it, when do we do it, what does it matter? Let's talk about how it actually happens in practice. And in just a couple of minutes you're going to see Tom actually walk us through an example of this. So we'll make it even more tactical. But these are the general, this is the general idea. First we generate realistic data with some known configuration. These are the parameters. I underscored realistic here because this is actually the place where people often go wrong. People often. And I mean, it's fine to start with less realistic data that's easier to recover, I guess, just to make sure that things are basically working. But the place where people go wrong is they're like, okay, I have a method. I know I want to use linear regression with an additive model. So I will generate data with an additive model and they'll just say, okay, you know, then the parameters will obviously recover or will recover as long as there's not a bug in the code. What you actually want to do is you want to think about, like without thinking about the method that you are planning on using for doing the statistical analysis, what is the most realistic data? Generate those very realistic data and then try to figure out which method or sets of methods work well for recovering the parameters of interest. So first we generate data, then we run the data through our analysis code. Then we check to see if the analysis code was able to find the true values that we generated in step one. And then at the very end we run this many different times, right? To make sure that we can do this consistently with different configurations. If you follow this process, this is actually like builds up a bunch of really good software engineering type habits where you have to separate your analysis code from the data. That's a really good practice, and this sort of forces you to do it. Additionally, once you do this, it puts you in a really good position because it makes the actual analysis that you're going to do very easy. You've already worked out all of the kinks and all of the bugs in your analysis with fake simulated data. And so when it comes time to do the analysis of your experiment or run the data through your model or whatever, it should just be a one button click to actually make that happen if you've done a good job of simulating realistic data. So these are all really good practices. They will make your life easier in the long run and they will also help you generate better models and do better causal inference. So for those who were here last week, we actually did this as part of the example, right? I was, we were talking at that point about which variables you should include, how you should choose variables that go into a regression. And I had generated some data where I knew what the correct answer was. And so in that example, we sort of walked through this backwards where I was sort of telling the story of doing an analysis and then figuring out that it was wrong. But we can actually think about doing this in the forward pass, which is to say the first thing that we do when we are testing some method, in this case linear regression, is we generate data. And in this case I generated data for basically, the idea behind this problem was we want to see how leg length and weight contribute to height in humans. So I had simulated some data where the parameters are known. I simulated leg, left leg lengths, right leg lengths, the weight, and then the height based off of those factors. So I know what the true value is of the left leg and the right leg contribution to height. And then we run two different statistical models to see how well they do at actually getting to that right answer. So here we can see that in the true model, like in the true data, the true answer that we want to get is that height is a function of leg length, but the average leg length times 0.45. And so when we actually went and ran this model, if we only include one leg length in an additive linear regression, then the model actually still recovers pretty well. And if you'll remember from last week, if we actually include left leg plus right leg, the model all of a sudden falls apart and doesn't work very well. At all. This is the power of doing parameter recovery is it helps us understand well, how right do we need to be about the true underlying data generating process in order for our model to actually recover the results correctly? So statistical estimation methods require trade offs. This is very important. No single method can answer every possible problem. You have to make assumptions, there's no getting around that. You should use the parameter recovery exercise to understand those trade offs. Where does it work? Well, where does it not? What additional assumptions do we need to make in order for this, this to be valid? When you're testing this, it's good to make sure that you understand in what situations your model will actually work and will recover the parameters. It's even better to understand in what situations will it not recover the parameter. Then you can start to think about, okay, can we test for these things to make sure that, hey, the data we aren't in this pathological situation that's going to cause our model problems. Cool. So I've been talking for a while. I'm going to hand the mic over to Tom who is going to walk us through an example in the context of media mix modeling.

Thank you, Michael. Let me just share my screen here. You should see some rstudio. Lovely. Some of you probably stare at this a lot. I certainly stare at it quite a bit. So yeah, as Michael mentioned, I'm going to walk through, I'm actually going to walk through two examples because I'm going to walk through a very quick one where it works well and then show you where it doesn't work so well. So just to orient us, I have some fake data here as part of our demo dashboard. We simulate a dashboard with fake data. And so we've already gone through the exercise of simulating spend data that has channels like meta prospecting, meta retargeting, mailers, podcast, linear tv, all of that good stuff. And so what I'm going to do is I'm just going to quickly subset this down to just a couple channels. So these first three metaprospecting, asc, Evergreen and retargeting and so look like this. And so I have three columns of spend. And what I'm going to do now is get some other data loaded here. I'm just going to simulate some data, right? So what I'm going to simulate here are the unknown rois. And we are totally ignoring several things here. We're not taking account of the base organic sales or not accounting for saturation curves. We're not accounting for time shift. This is a super simple Model, just three spin variables, no non linearities, no convolutions, no Bayes sales, nothing. And what I'm going to do, to start, I'll explain what this function is and what some other stuff that I passed over. What I'm going to do to start is I'm simply going to set a single number so somewhere between 0 and 10 as the ROI. So I've got this coefficient matrix, it's indexed by the number of time steps, which is 810, number of spend variables, which is three, all zeros for now. And what I'm going to do is I'm going to fill it in with a random number and it's three, four and a half and nine. These are our coefficients. And so in this particular case the coefficients are all constant, right? So no time varying effects. And I'm going to simulate what the outcome must be. And because spend varies, so does the outcome. But I want to add a little bit of noise and I've decided I'm going to add 5% noise to, to this sort of like raw prediction. And so my actual outcome is going to look a little bit different just so there's no singularities in the, in the linear system. So I have my unknown parameters, I have my spend data here, and I have my simulated outcome, which is this. And so now I've simulated the data and here what I'm going to do is I'm going to estimate the model. And so as Michael mentioned, we're just using a very simple linear model, right? We have a linear data generating function. It satisfies the assumptions of the model, right? And so we should get results that are pretty close to the data that we generated. So I think we had, I plotted my coefficient matrix. One of them was a little above three. Cool. This must be it. One was between four and five. We get pretty close. One was a little above nine. We get pretty close. And so if I plot my coefficients against their estimates over time for each of the channels, this is channel one, channel two be channel three. We do pretty close. So what we've done here is we've simulated data that satisfied the assumptions of the model we were using to estimate the parameters. And then we estimated the parameters and then lo and behold, we got a really good result. So phenomenal, right? If life were this simple, we would be super happy. So let's start to complicate it a little bit. So up here I have this thing called a kernel. You might be wondering what it is. It is the beating heart of a Gaussian process, and it will allow me to simulate data if I do this matrix multiplication from a Gaussian process. And a Gaussian process allows you to generate curves that sort of smoothly vary over time with seasonal variation. And so if I just run this a bunch of times, you can see, like, the kinds of data that a Gaussian process might generate. And then I have this function here called bounded rois, right? So what I want to do is I want to simulate data from a Gaussian process, right? So it smoothly varies over time. It might have some seasonal patterns, but it's going to follow the assumptions that today is like yesterday and today is like a year ago to kind of believe. But in addition to that, it's going to stay within some bounds that we sort of generally know to be true. So in this particular case, we're going to say the ROIS can't go below zero, right? That may be something you believe. And they can't go above 10. And so when I simulate data according to this bounded ROIS function, it's going to look like this. So we can see it sort of over time. ROIS might go up or down, but they're going to sort of stay within this 0 to 10 range. And by playing with this parameter, I can make it almost flat. You can see the Y axis is collapsed to a very small value here. And by playing with this value, I can make. I can sort of suppress the ROIS and bring it down a little bit. Right. I'm not going to be doing that, but just to give you an idea of the kinds of things you can do to simulate data. So if I simulate data like this, I'm going to get three curves, right? So previously it was like three straight lines. We were making the assumption that the ROIS were just flat over time. And in this particular case, we're saying, okay, the ROIS can change over time, and this one starts low, comes up, and then varies over time. Now, it might seem unreasonable, but remember that we're looking at more than two years of data here, right? So this is 810 time steps. That's two years plus 90, I think, is what we generally do, or something like that. And so having these sorts of estimates that change this much over that time period is not actually unreasonable. And so we stimulate data the same way, and I get some time series of the outcome, and then I estimate, wait, yeah, okay.

And before we look at the results, I want everyone to sort of pause and imagine like, or think about, what do you expect is going to happen? Here we go. So we generated potentially realistic looking data, right. Of how marketing efficiency is changing over time for these different channels. And now we're estimating a model that you know, doesn't have time varying covariates, estimates, sort of fixed covariates. What do we expect that we're going to see? And you don't have to share what you're thinking here, but you should pause and think about this. What do we expect or what do we hope the results might show? I would maybe hope that we would get the average effect. Right. So even though the coefficients are changing over time, the true coefficients are changing over time, the LM estimates are going to be at the average in the middle there. That's what I would hope prior to having done this result where I, where I saw what the analysis was. But so you might want to think about like what do you expect we're actually going to see here? And Tom, I will hand it back to you.

Does anybody, I kind of accidentally showed the results. So if anybody didn't see them before making this and wants to actually audibly guess, you're welcome to right now for about the next 10 seconds before I do show the results.

Let's show them. Let's show the people what actually happened here.

Cool. Yeah, so I mean let's find out so. Well, right off the bat one of them is negative and that can't possibly be true. I didn't save the seed so I'm just sort of rolling here. We can run this a couple times and see though in this particular case.

Does look that one looks pretty good. That one looks pretty good.

That one looks pretty bad. Way off and then that one's way brutal. Brutal. And you know, typically with these sorts of parameter recovery analyses you never know. You could have just gotten unlucky. So you know, you want to run it a bunch of times and we're sort of never getting close on this channel. That was not so bad. Can take a look at what it looks like for some of the other channels. So good. So I think not so bad. At any rate this is generally you sort of get the idea right. We, we generated model data according to, you know, a very simple linear regression and things seem to work and then we complicated a little bit introducing time varying coefficients and now all of a sudden the model is completely unable to recover the parameters in any reliable way.

And I think, you know, that's exactly right and I think let's talk. But I think it's, it's important to even step back and Think about what does this mean in the context of a business problem? And if you're doing this in the context of a business, and especially in the context of like, media measurement, it means that you would make a terrible recommendation to the business. Right. Your. The parameters that are coming out of the model do not match reality at all. And they are going to lead you astray. If you followed the results of the LM regression, which again showed all of the results were statistically significant. R squared, I'm sure was probably not great, but like, fine. You know, there's no obvious reason why this isn't working. We've got statistically significant results here. You know, it seemed to work, it produced results. We got numbers, but they're just totally wrong. And if we follow these, this, these results, it would totally lead us astray as a business and we would make the wrong decision and we would invest in the wrong channels. Right. This is a good example. The one that's on the screen. The result of the regression says that this is our best channel, 11x ROI. Actually considerably worse than that on average and especially in the recent time period. And I don't. Can't ever remember how to draw on the screen. But in the last couple of weeks, it's especially bad. Right. It's one of our worst channels. So these are the reason why we do this is because we believe that this can help you make better decisions and avoid these sorts of mistakes in the work that you're doing. Cool. So this was all of the content that we did cover. Tom, you did an amazing job on the live coding. So thank you for walking through that. That was great. Now we have open time for Q and A. I think we've got. How much time do we have left? 15 minutes. So we've got 15 minutes for open Q and A. What questions do you have? We are happy to talk about parameter recovery specifically or also any other interesting statistics or modeling questions that you might have that you want to discuss today. Dummy data example. Let us. I believe the answer is yes, we can share out some of the. We can share this out. Let us clean up the code a little bit maybe before we share it out, but we should be able to share out something along these lines that y' all can play around with. How are we feeling on under. Well, maybe since I'm trying to, you know, try to generate some, some engagement here. How many of y' all have done parameter recovery analysis before? Feel free to drop in the chat or use one of the cool little reaction icons. Ben says nope, Nazar, I'm going to take that dash as a no. Mark has done it, so that's great, Mark, props to you. Is that hand raised, Mark, a question or a comment or maybe, Mark, do you feel comfortable sharing where you have done parameter recovery before? Sure, yeah. I work in experimentation and we, the other data scientists and I before testing out new model architectures will part of like the verification before deploying do the same sort of like simulate on known parameters, recover, verify that it works, and then start checking against experimental data. Love it. I think the other place that people often sort of come across this idea is, is in power analysis. So if you've done a power analysis before, that is effectively a parameter recovery exercise. You're going to generate theoretical experimental data with different population sample sizes and then see if your test can recover the results or not and make sure that you're targeting either some ideal power or some false positive rate or true positive rate, etc. And there are trade offs there. But the idea of doing a power analysis is very much a parameter recovery and that's often a place that people first encounter this idea. But if you're doing experimentation methods like Mark, I think it's a really important part of the workflow to make sure like does this thing actually work when we know the answer? And again, you would probably be surprised at the amount of times that it doesn't actually work the first time that you run it. Jeff has a really good question here. Tom, I might kick this over to you. Or are we out of time now? We've got, we've got time. We've got time. We got, we've got time. So Jeff's question is about how do you generate data? How do you use that domain knowledge? Like we've got ideas, you know, I believe some things by, you know, I don't know, domain expert partners maybe believe other things. How should we think about doing that?

Yeah, I mean, so there's another step in the workflow for building a large, complex statistical model that we sort of covered, sort of didn't today, which is called a prior predictive check. Right. And so often when you have a complex statistical model, it'll be very hard to get any good information from anybody about what the little knobs and dials of the parameters are. So, you know, for example, in our case, one of the things that we have to estimate is when will this channel saturate, right? When will it stop being able to perform? And you'll reach the sort of marginal, declining marginal efficiency of spend in that channel. People might have ideas, but they don't really know. And it's hard for them to give you any guidance there. And this is talking to marketers. So even in an area where they ostensibly might know something, it's very hard for them to give you information. Similarly, with things like timeshift, right. It's like a deep parameter in the model. Some are hidden in there. They might have trouble sort of really telling you what they think. So typically, it's hard to inspect the parameters of the models themselves and get people to give you good information about what they should be. But typically, what you can do is you can generate that fake data and, you know, using the same approach that I did today. And if the outcome variable simply does not smell right or looks way off, then you know something about your priors. The parameters that you're simulating must be wrong too. Right. So, for example, to use Michael's, the, you know, example that he put on the screen earlier, you're building a model to explain height. If you put priors on the impacts of nutrition, the impacts of, you know, caffeine, or weightlifting on people's heights, and the coefficients had such an effect that people. The prior predictive range was anywhere between negative 10ft and 30ft, well, then you would know that something must have been wrong about those priors, because your knowledge about people's heights is that they generally range between 4 and 7ft as adults at least. And so really, using that prior predictive check, which combines all of the information in the model that you've assembled and produces the outcome, which you generally have some knowledge about, and calibrating the priors so that your outcome is reflecting your existing knowledge, that's the best way to do it.

Yeah, And I mean, I would just say that, like, you should spend a lot of time talking to experts and reading the existing literature and whatever field that you're operating in, if you're doing marketing science, you know, there's been lots of books that have been written, lots of articles that have been published. There are, you know, the folks at Wharton who do a lot of amazing marketing science research. There are the folks at the Ehrenberg Bass Institute in Australia that do a lot of amazing marketing science research. Like, you should really be getting into that literature to understand what do we know about how marketing works based on the studies that have been done. Combine that with talking to people in your organization, again, talk to marketers. How does this marketing channel work? How do you buy the media? How do you, you know, what do you believe about it? Talk me through it, that those sorts of conversations are critical to doing this sort of work correctly. And again, the more observational the data are, which is true in the case of media mix modeling, the more important this stuff is. When you have more experimental work, you can often make fewer assumptions and have simpler models. When you're doing more observational work, you really need to make sure that you're getting the data generating process right and that deeply understanding the field in order to be able to do that correctly. And then Jeff is clarifying on the process here, and I think that. So, Jeff, I think the answer is right, so I'll read this question aloud. So we do a lot of simulations of ranges for priors in our prior predictive check and see how often we can get close to the outcomes with our small test. I think that's basically the idea. So if you are working in a fully Bayesian framework, which some of you are and some of you may not be, it's totally fine. You don't prior predictive check works, or, sorry, parameter recovery should be done no matter what framework you're working in. If you're working in a Bayesian framework, you can take your priors and use that to sort of automatically generate data for doing the parameter recovery check. Like a Bayesian model works both forwards and backwards. You can use it to generate data, or you can use it to recover parameters. And so it makes that workflow really nice. But that's basically the idea is if you believe that your model actually reflects the real world, and this is a thing that you should inspect very deeply before you get started, you can use that model to simulate data and then verify that your model can actually recover the parameters from that simulated data.

A couple people have mentioned Richard McElroth, who places so much value on these exercises that he once said that the minimum effective sample size is 0. Meaning that, like, you can get so much value out of just simulating data from the model without actually having any observational data. Yep, we all. Michael has it. It's our, like our. Our bible. So we're extremely big Fans of Richard McKelworth over here as well.

All right, any other questions? Anything that's been burning for y'?

All?

Again, anything that we can help with? We've got eight more minutes, and we just want to be as helpful as we can with this time. Ooh. Any major gotchas along the learning path for this? I would say not really. Honestly, this will. This is, I think, the best way to actually learn statistics. I think all of the Classroom learning that I did as like an undergrad did not actually teach me how to think statistically. It taught me a lot about like, okay, there are some formulas you can learn. This is the formula for the standard error or whatever. And there are some computational methods that you can apply. This is how you type in the code for Stata, this is how you interpret the results on the other side. But I didn't really understand any of it. Doing this sort of simulation work will really help you understand what is actually happening under the hood and how does it actually work. And so I would say that most of the gotchas actually come in the other way of learning statistics. And this is probably the simulation based way is probably the best way to actually learn statistics. And I would say just try it again. If you're only working with simulated data, you can't really go wrong, right? You can only understand you're not going to show these results to anyone. What you're going to do is you're going to learn a lot about how these methods actually work. And also, you know, you'll probably become a better programmer because like simulating data is in many cases like simulating realistic data is often the hardest part of doing this. Jay is asking about model evaluation and best practices on non simulated data in particular. Tom, I don't know if you want to take a quick stab at how we think about that.

I mean the, the most, in my opinion, the most important thing that a statistical model can do is help you predict unobserved observations almost always. And so the there's a number of dimensions along which you can evaluate the performance of a model before you expose it to live data. You know, you want to see if the priors make sense, prior predictive check, you want to make sure it can recover parameters that are compatible with the model. Parameter recovery check. And then once you have the data, you want to do a couple things. But the first thing I would look at is, okay, if it's a time series model, show it all but the last 30 days and can it predict the last 30 days? Show it all but the last six months and can it predict the next six months? If it's not a time series model but it's say a customer level model, you know, show it all but 10% of the customers in it, can it predict what the remaining 10% of customers did? Did they click on the promotion or not, whatever it might be. Because if a model is actually working and it's capturing something fundamental about the data generating process, it should be able to predict new data. So that's the most important thing. There are some other things that also matter for us. Stability matters quite a bit. So you know, we update the model every week. So every week we're appending seven new data points and chopping off seven from the beginning and then re estimating the model. The model should stay relatively stable. Obviously some things are going to change because some of the data changed. But if we are wheel jerking our customers every week, that is not great. And so robustness to data is also a check that we use. But if you're going to pick one out of sample, predictive accuracy is the most important thing.

And then Jay had a follow up question on what metrics are you using to evaluate the predicted data? Maybe you want to talk about CRPs.

Yeah. So we use, so there's a lot of different metrics that one could use. We use something called the cumulative rank probability score. The reason that we do is because we use a Bayesian model that doesn't produce, I don't know why the thumbs up happens, but it doesn't produce a single estimate. It produces 500 estimates. Right. And things like the mean absolute error, mean squared error, don't generalize well to an ensemble forecast which is like what we're doing or forecast from an MCMC model. So this score, it's called a proper scoring rule. There's a whole literature on forecasting accuracy and proper scoring rules, but it gives you credit both for having a narrower forecast and having a less biased forecast. And so that's the metric that we use. So if you want to look it up, it's called the cumulative rank probability score. It's perfect for this situation. I'll just quickly say shocks and change points. Yes, we do. I didn't include it in this simulation, but actually I would say that the data was a little bit unrealistic because it didn't have enough big spikes. And most of our customers data you see, you know, Black Friday, Christmas Eve, New Year's Day, whatever it is, Valentine's Day, Mother's Day, there can be massive spikes, especially alongside promotions as well. We have, we believe, we're not really sure if other MMs really have the kind of spike modeling technology that we do, but we have a really, I think, cool way of modeling both the additive effects, but also the cannibalization that spikes have on either side. So people delaying their purchase to take advantage of a promotion or accelerating their purchase to take advantage of a promotion. So you see a lot more sales on the day of than really the spike added. So yes we we do include that in in our models.

All right, we just have two minutes left so want to wrap up here. Thanks everyone for a great session. This was really fun for us. Lots of really good questions. For the questions that we didn't get to in the chat, we're gonna save them and we will follow up over email with our answers. So really appreciate all of the engagement. If you aren't already, follow Tom and I on LinkedIn. We are publishing a bunch of content about these statistical techniques and really how to think statistically and how to do causal inference the right way. So give us a follow. We'll continue posting there and then we'll be doing more of these office hours. If you have ideas for topics that you want to see us talk about and cover, we would love to hear it. We think this is a fun format with a little bit of a bite sized lesson and then some Q and A. So hopefully you found this useful. Give us the follow, let us know if you have ideas for topics, feedback, thoughts. We really appreciate it and otherwise have a great rest of your Thursday and a great weekend. All right, thanks y'.

All take care.