What are lift tests and where do they fit with an MMM? | Recast Office Hours #5

Foreign.

I think it's about time to jump in, so we'll start ripping through it here. So, for those who don't know, I'm Michael with me and my co founder Tom. We both co founded this company, Recast. We've had backgrounds in statistical inference and a passion for causal inference. In order to keep this organized, we generally have a lot of people in these office hours, so please stay muted unless we call on you and then we can bring you in to ask a question. Writing questions in the chat is really helpful. So Tom is going to be keeping an eye on the chat, answering questions where he can and servicing really good questions to discuss as a group. Keeping them in the chat is a great way for us to see and parse through everything that people are asking about. Any questions we don't cover, we will answer offline and share out via email.

Thanks everyone for coming.

Really excited about this. Today we're going to be talking about lift tests and calibration of a media mix model. Tom and I have prepared a lot of content to talk through. This is more theoretical content because a lot of these problems are really challenging. And then we'll open it up for Q and A at the end. As always, you know, this is not a pitch. We're going to talk about how we think about these problems at Recast, how we have solved them at Recast. But really the reason for talking about them is so that, you know, you as practitioners can go and implement this. So definitely raise questions. We'll be talking about, you know, some of the things that we have thought about and some of the hard problems here, but really not designed to be a pitch, designed to be helpful for all of y' all who are out in the world building models, thinking about the right way to do marketing measurement. So jumping into it, a couple of key points, key takeaways that we want y' all to take away from this discussion. So the first is that all of the methods that we're going to be talking about are approximating the truth. So lift tests, mmms. No one has a monopoly on the truth. No one has access to the actual real truth. They're all methods that we're using to try to learn something about the real truth that exists out in the real world. But they're all going to be approximations, they're all going to be noisy, they're all going to have uncertainty associated with them. The second is that time shifting effects have a lot add a lot of complication. When we're talking about, thinking about comparing the Results of a LIFT test with the results from a media mixed model. That's really, really important. Correct handling of uncertainty is really critical to get right. Again, as you're thinking about trying to do this, four time varying estimates are really important and we'll talk a little bit about that. And then last we'll talk a little bit about in theory, how you can use Bayesian methods to incorporate the results from LIFT tests into a Mediamix model. All right, so I want to talk about time shift effects and I'll maybe do an overview here and then Tom might chime in with something. So I put this video, this image on the slide. It's from a really great blog post that was written by um, Gu Feng at On the, on the Meta Marketing science team. Guang is the original author of the Robin package. They've thought a lot about incrementality testing, lift testing and the right way to calibrate mmm results with that. I don't know, Michael Solomon, if you can find that blog post and drop a link into the chat. But it's really, really good. Highly recommend that everyone who's, who's thinking about this problem go and read it. But I want to talk a little bit about what this image is showing so that we can think about what is the real problem that we're trying to solve. So looking at this image, we've got these black lines which are showing the start of the list study and the end of the list study. We've got sort of an illustration of the test and control group here. And then we've got some illustration of the effect that we are observing. Right. And the problem here is that there's areas A, B and C C is carryover impact. So we ended the test on Sunday, but the marketing effect doesn't actually end on that day. It probably has some carryover. And so there's something potentially that we're not observing just because of the way the test is structured. There's size A which is carryover from previous periods that we need to be careful about. Right. Because marketing from previous periods, that's potentially having an impact on the current period. And then there's B which should be the lift generated by the test during the test period. And so when you're thinking about, first of all analyzing the lift of your, analyzing the results of your lift test, you need to be thinking about A, B and C and which parts of those are you capturing and which parts are that you're potentially not capturing. And then when you're calibrating your M and M model. With those results, you need to make sure that you're calibrating to the right value. Tom, any other thoughts on this?

On. Specifically on timeshift? No, unfortunately, it only gets worse from here. But this is a pretty. This is a great graphic and a good visual for how timeshift can be a complicating factor when it comes to using lift test to calibrate an mmm.

So the core point here is that lift tests do not produce certainty, right? What's illustrated here is just again, standard lift test. There's some control group, there's some treatment group, there's some incremental revenue or conversions that are observed associated with the test. And what we get out of this should be some range of values that are compatible with the data that we have observed and some lift that we're seeing. And so a problem that we. That I often see with, you know, our clients and people who are coming to me asking me for invites, they'll say, hey, you know, we ran this lift test. It showed, you know, 3% lift. And I'm like, I have no idea if that's good or bad or anything. Right. In order to correctly interpret the results of a lift test, you need a couple of different things. You need to know how much lift was driven in the test, right? But you also need to know how much spin did it take to get there. Because it could be that 3% lift is really good. That's maybe one of your best channels. One of your best channels, you know, by. If you spend $10 and you get 3% lift in your overall revenue, that's amazing.

If you have to spend $10 million.

To get 3% lift, then that's really terrible. So you need to understand how much investment did it take to get to that lift. And then you also need to understand the uncertainty balance associated with that estimate.

Right?

Just saying that it's statistically significant is generally not enough. Because when people say, oh, it was a 4% lift and statistically significant, what that generally means is it was 4% and not zero. But what we actually care about is what is the range of intervals or what is the range of values that are compatible with the test. So what I like to see from lift test results is, hey, there was a lift of X ranging from Y to Z, and that's associated with a CPA or an ROI of alpha to beta, right? And so you want to understand what is the distribution of results in the test. Some tests are really highly powered and will have a very narrow confidence interval. Unfortunately, in the context of marketing, it's very difficult to get enough power because marketing has relatively subtle effects. And so often, even when people are running really well run lift tests, especially if they're doing some sort of geo lift testing, often the uncertainty bounds on, on the estimates are very wide. Right. It's not uncommon for people to go run a lift test, be like, okay, yeah, we ran the lift test, $150 incremental CPA. That's great. But the confidence bound ranges from, you know, $40 CPA to a $900 CPA. And it's important that you're really reporting out on all of that information. And when it comes to actually using this data to calibrate an EM mix model that we are using that full range of uncertainty, not just the point estimates. Thomas, anything to add to that?

Yeah, I mean a couple of things to add. So one thing to add is that, and this is getting to like, if you do what we do, which is we use liftest to calibrate mean emergence models, is that you have to be very careful about what the summary statistic is. And typically the summary statistic is overall effect over a time period. So like you might run a test test for a week and you're talking about the average RLI or CPA or LIFT or whatever statistic you're using for all of the spend during the week. And it may not be the case that your MMM actually has an easy way to impose a constraint on a sum of spend. Right. It may have like a constraint for each day of spend or a month of spend. It may actually be difficult to translate what the summary statistic is to the actual underlying values that your MMM is producing, which may be an ROI per day or maybe an ROI per month, or maybe a single ROI across the whole time period. So that can introduce a mismatch between what your geolift or your experiment is estimating and how to actually use it in the context of an mmm. Second thing I would say is, you know, we got a comment here by er Baver that the, this percent definition of lift can be misleading. I would say it's all misleading. These are so many different definitions out there. Often you will see percent lift, which is a percent over baseline revenue. I don't particularly like that. I like it in terms of ROI or CPA, which is actually not quite what you have here, Mr. Or Mrs. E.R. baver. But regardless, you have to translate whatever the statistic is into something that you understand, are comfortable working with. Typically we do see much wider uncertainty Intervals than people are generally comfortable with. And that has the paradoxical effect of leading people to just ignore them, right? So often what we see is like, oh, the uncertainty intervals are super wide. So I don't even look at them, I just look at the mean. And that's exactly the opposite reaction that you should have and your stakeholders should have. They really should be accounted for, the uncertainty. And the wider it is, the more important it is to really understand what is the range of plausible values. And then finally, the last thing I'll add here is that units matter, right? So almost every statistical model is going to be in terms of KPI equals effectiveness time spend, right? And that is, that is effect equals alpha times spend. Now, if you rearrange terms, you'll notice that CPA is actually an inversion of those terms. And CPA is spend divided by effect. And what that means is, is that your uncertainty and the effect can really blow up the uncertainty in the cpa, especially as the effect travels towards zero. It means that the CPA tends towards infinity. And so it could be really common to have really, really wide CPA intervals. Because this, what's happening is, is that the effectiveness of the channel, the uncertainty of it includes values very close to zero. And that's going to blow up the CPA to very, very large values. And so that's something to be aware of too. That can almost overstate the uncertainty because if the uncertainty gets close to zero, your CPA can look like it has a really, really wide uncertainty. But all that means is that the un. It includes values that are close to zero. So that's just another wrinkle that we see all the time. That is something that's important to keep in mind.

Awesome. Those are all amazing points. Makes a lot of sense. The other thing to keep in mind is that experimental results are generally time bound to the time that the experiment was run. So this is a way that we like to think about how you should, how you should think about how do we use these results practically. And so during the test read, you will get some read on the performance of your different marketing channels and their incrementality. And those results apply to that test period. And then what could happen is before the test, right, things might have been different, might have been more performant or less performant. After the test, things might be different, the channel might be more performant or less performing. And so as you move in time away from those test results, you want those test results to have less of an impact, right? You don't want to run A test in one channel, one time and say, oh, this is the true incremental ROI of this channel for a right, that is one sort of sample, one snapshot in time. And so it's really important to think about the test that way. And again, this is true for using those test results for decision making independent of calibrating to an mmm. But when calibrating an mmm, it's especially important to take a about it this.

Way and I'll just add, I mean this is one major problem with MMMs that don't have time varying ROI channel efficacy. Because it can be really common for one of our customers to have run multiple lift tests in the same channel, kind of like direct mail. A lot of our customers are literally running a matchback holdout test every time they drop direct mail. And so if you are not, if your model can only have one ROI estimate per channel, it's actually totally plausible that you're not going to be able to cover the uncertainty even of any one of the lift tests because some of them might be high and some of them might be low. And if the model is only forced to find one answer, it might just draw a line right through the middle of them. In fact, that's probably what it's going to do. And so, you know, this isn't an office hours on time varying coefficients and mmm, it's just one. Just to illustrate why it's so important. Especially if you're doing multiple lift tests in the same channel, it is very unlikely that they're going to land on the same result. And if you don't have a model that can incorporate that, it's going to lead to some problems.

Yeah, exactly. And so this slide sort of illustrates what this often looks like for companies that are that have a really overly strong lift testing and experimentation program is you'll run tests at different times, you will get different results, they will have different amounts of uncertainty associated with them giving us different reads on how this channel is performing. Some people that I talk to are like, oh, it's the testing that's bad. And you know, really like the actual truth is just right in the middle. I actually don't think that's true. I think that in some cases it probably is, especially with very noisy tests, like if the uncertainty bounds are really, really, really wide, like it might just be noise or the point estimate will be very noisy. But when you're actually taking into account the uncertainty, I think a lot of times what we see is that true channel performance changes over time. There are a Lot of good reasons to believe that that's true. And so when you're drawing these, when you're running these experiments at different points in time, what you're actually really seeing is how is the channel performance changing? So that's what we see here. And again, this is very, very common. And so you want to think about how are we going to calibrate the mmm with these results that are potentially changing over time from these different. So this is a place where the time varying coefficients work really, really well. And so we're again, we're not going to talk about like how to actually implement this in on your own. We have a bunch of docs@docs.get recast.com if you want to go check out, check it out and see how we've done it. But the idea is that the way that we recommend that, that brands configure recast is first of all, you start with every channel having uninformative priors right around the ROI. What's the range of plausible values from 20x ROI to 0x ROI? Right. And that's the same for every different marketing channel. You don't just want to set that again as being like, you know, the range of plausible values are the values that came from the most recent lift test. That would be a mistake. So you want to start with the uninformative priors that cover the full space of values that are, you know, reasonable for your business. But then what you want to do is you want to use the lift test priors to constrain the space only in the time when the test was actually run. And that's what we show in the bottom diagram here. And so this is a really nice thing about this Bayesian approach, is that you could have wide uninformative priors for the system as a whole. And then when we have specific experimental evidence, we can use that as a prior in the model to constrain the values that the model is compatible with, but only during the time that the test was actually run. And so you sort of have two layers of priors that are happening here and those working together effectively force the model to be consistent with the test, but also have these other properties that we care about, which is the weight of that test will decay over time. So if we ran the experiment two years ago, it's going to have very little impact on what's happening today, but the model is still going to be consistent with our overall uninformative priors as well as that that test in particular pause For Tom, I don't have too.

Much to add here. This is a nice little visual about how we do it. Obviously this interacts very strongly with the time shift things that we had at the very beginning. So you're sort of like, you need to decide where you're setting the priors. Are you setting up setting the priors on the day of spend? Are you setting the priors on the length of effect? Our model is calibrated to set the priors on the day that you spent the money and then indexing the effect to that window. And so it's sort of by default trying to omit A and include C if you can remember the graphic at the very beginning. But that is a wrinkle that you have to account for here. Even when you're setting it for a range, you have to account for the spillover effect both into the window and out of the window. Not too much to add. It's just that that complexity bleeds into.

How we do it here. And I think. So I just realized that I forgot to actually add a slide for this. I'm going to add a blank slide here just so that we. Oh, no, I don't want to do this. Okay, great. So, Tom, the thing that you wanted to talk about that we don't have a slide for here is saturation effects and how to think about those. I don't know. Do you want to talk a little? Do you want to riff a little bit on that?

Totally. There's actually a couple of different additional wrinkles that you could sort of imagine having. I was just answering, Kevin. I saw your question. I'll get back to you in a minute. So saturation is an interesting one to think about because typically with a geolift test, let's imagine you have a market that's the U.S. right? And you run a really simple geolif tester. You split it by like one half, you know, east of the Mississippi and west of the Mississippi, and east of the Mississippi is your test group and west of the Mississippi is your control group. Obviously, any GLF test is going to be more sophisticated than that. And you spend, call it a million dollars, eastern Mississippi, and you get an ROI of 2. Well, that does not mean that nationally you should expect an ROI of $200 million. Nationally, you should expect an ROI of something above 2, because you're actually saturating the channel substantially less when you're applying all over Mitchell, you've got to go down. You're. Nationally, you are saturating the channel substantially less when you're spending across both markets than when you're only spending in one, right? And so this can become a problem when your test group is a meaningful proportion or actually the proportions that matter, it could be a problem even if it's a large fraction or a small fraction of the overall market. You have to account for the saturation effects, right? You would expect the overall market to have a higher ROI at any given spend level than the spend level that you put into the channel when you're just spent into 20% or 30% or 40% of the market because you're not saturating the channel in the same way. Now, some kinds of tests don't have this problem, right? Like direct mail. When you're doing a matchback holdout test, it's not going to have the same problem unless you are, you know, going further down the list of prospective customers and it's saturating that way. But this is an important thing that you have to bear in mind when you're extrapolating the test results to how the channel is going to perform nationally. You have to account for the fact that you've only saturated some of the channel when you are doing a GLF test and you're only spending a portion of the market. Tom, the other, and this is not related to saturation, but the other thing that matters is lower funnel effects. So one piece of the puzzle that is typically completely omitted from lift tests is induced spend, right? So when you spend on upper funnel channels, which could be linear tv, it could be Facebook prospecting, it could be podcast, typically that's going to have a complicated downstream effect that's going to lead to both more sales or more acquisition, but it's also going to lead to more spend. Why is it going to lead to more spend? Because more people are going to be searching for your branded keywords on Google. More people are going to be coming to your site via affiliate sites where you'd have to pay a commission. And so in addition to increased sales, there's always, almost always going to be some amount of induced spend as well. And when people look at lift tests, they're only looking at the quantum of spend and the quantum or the estimated quantum of sales, and they're going to be dividing one by the other. But that quantum of induced sales is going to be inclusive of the indirect sales, right? So if somebody, if you showed an ad in the test market and then that person got interested in your brand, searched Google, saw a brand new search ad, clicked on it, so you spent a little Bit more meant eventually bought. The eventually bought is going to show up in the effect term in your direct spend is going to show up in the spend term. But what's going to be missing from the equation entirely is going to be the induced spend on branded search. And that's actually pretty important because that can take a channel that is one of your best channels to be one of your worst channels or one of your worst channels be one of your best channels, depending on how it's accounted for. Because sometimes you could be driving spend into relatively inefficient channels like affiliate commission can sometimes be extremely inefficient and just be totally cannibalizing sales that you would have made anyway. And some channels can preferentially drive into those lower funnel channels and some channels don't. But you're not going to pick that up in a live test unless you're measuring it as well as an outcome of the test. We've seen nobody do that. Nobody's even thinking of it. But one thing that I would love if with tests move it in this direction is not just measuring as an outcome the the effect of the channel on sales, but also the effect of the channel on lower funnel spend because that's a critical part, critical component of the overall efficacy of a channel. That's totally missed with methodologies today.

Great, great points. That's basically all of the the sort of content that we have for today. So this stuff is really tricky. We talked today about a bunch of the problems that you need to be thinking about as you're thinking about doing this. Sort of one thing about the lift, interpreting the lift test generally, but then particularly when you're including the development in MLM and using them for calibration. We have a bunch of information on how we think about doing this@docs.getrecast.com but in the meantime, we would love to open up the floor to questions so we can talk about anything that y' all are interested in, whether it's how to do this internally and more questions about how we actually operationalize this or basically any way that we can be helpful or I don't know if there are good questions in the chat, Tom, that we should serve us.

I just answered the one that was remaining. So folks have questions or things that they want us to dive into. Don't be shy. You can speak it out loud or dump it in the chat. Either way, it's totally fine.

Shana, for your question, do you mean in AMEND vendors?

I would like to ask a question if it's appropriate.

Yes.

Yeah.

Quick introduction, one sentence introduction. I'm Ernest Bader, I work for as a senior research scientist for Amazon. We often talk to third party vendors who are partnered with Amazon with their MMM estimates and we look at multiple causal estimates. The question is actually probably open ended but related to what you described based on observations available within Amazon APIFunnels. Randomized control trials often show very little incrementality, short timescale. So typical randomized control trial at Amazon has two week treatment window, two weeks conversion window. So like for upper funnels it's challenging to measure it. Randomization on longer time scale is also obviously challenging. It's hard to maintain randomization or even preserve identity. So IP addresses, etc. So like I would be curious to hear your intuition about that. It's clear that MMM model structure allows you to have low incrementality long timescale but show reasonable roi. If you look at longer time scale. But your intuition about this would be extremely valuable.

Yeah, I think this is a really interesting question. So just to make sure that I understand the question that you're asking is. Well, I'm not sure that I fully got exactly which the question is so, so I'll ask the clarifying question. So you're asking about, okay, there are upper funnel channels that have these very long time shifts. Only a fraction of that is going to be captured in any lift test. And then what's the subsequent question?

Exactly right. So if you look at upper funnels effect in randomized control trial like someone can watch TV but as you actually mentioned in your presentation, it might induce spending in other channels, et cetera. So basically the point is, which I believe you also like implied implicitly definition of ROI does depends on the time horizon where you do measures that on the other hand your lift experiments or causal experiments, they usually done at very short timescale. Right? So like and MM provides natural bridge to glue these two things together. But your insights, how you view it, that's what I would like to hear from you guys.

Yeah, I mean this is a place where I think it's important to think about this chart, right? Which is like what is the, what are you measuring with the lift test and what is your lift test or what is your MMM potentially measuring or like what's truly happening and then making sure that you're calibrating those appropriately. One thing I will say is I think that there are a lot of people who make claims about in media mixed models ability to measure really long time effects and I'm very skeptical of those. What I have seen is that a lot of those methods that I have seen, they're just really noise or they're only driven by modeling assumptions. They're not actually modeling it. They're just assuming that TV has a, you know, an effect over six months and they just multiply the first month effect by some number to get it there. And like that is I guess okay as long as you're transparent about that assumption. But I think often when it comes to effects that are coming 90 plus days after the marketing activity has happened, there's just, it's very difficult to get a read on it either with some sort of control trial because they're just operational complexity or with an MMM just because of identifiability reasons. It's very difficult to identify that parameter as it should it be zero or should it be some high number? And I have not seen anyone be able to do that convincingly that has like an identifiability check. So broadly I don't think there's a good answer. I think, you know, to the extent that people can, running longer tests is really good. I think there's a lot of good theoretical reasons to believe that there should be sort of a reasonably like some drop off in the effectiveness over time, like most of the effects should happen within the first week and after six weeks you're going to see some dropping off effects. And so running an experiment that can help you see what that is and then just fitting a curve I think is a reasonable approach. But broadly I don't think there are good answers here. Tom, I don't know if you, if you disagree with that, but that's like my thought on this.

I don't disagree. I think when you are talking about really long term effects you are sort of darn if you do, darn it if you don't. I mean like that the real problem, ideally you would run a really long lift test, right? You would just say okay, we're going to have these test and control markets and we think there's like a 90 day like effect window. And so we're going to actually keep the test and control market separate for 90 days and measure the effect of the, of the channel. The problem is is that operationally that's extremely difficult to do because you're, you typically need results faster than that. There's also tons of spillover effects, both actually geographically spillover effects and temporally and it's really hard to keep everything else constant when you're running a test that Long. I think the best that you can do is use timeshift like density estimates, like what is the shape of the timeshift curve to try to get earlier results before the, you know, full effect window is done. What I mean by that is, is that if you can be pretty confident, and I don't know how you would be, but if you could be pretty confident that call it 15% of the effect showed up within a week, then you could get an estimate of the effect within a week and then scale that up by dividing it by 0.15. Right. To get the full effect over the full course of the, you know, effect window for that particular channel. Now, of course you're layering uncertainty on uncertainty here. And so, you know, at some point you're just going to end up with sort of like a, an overly wide uncertainty interval that's going to be hard to use to begin with. But I don't know how you get around doing something like that in order to actually run, run a test on a very, very long spect window channel. That's just very difficult if, if it's okay.

One last quick question related to the previous one.

Do you see clear.

So in terms of ad stock transformation, do you see a clear pattern, let's say in terms of, in terms of, sorry, ad stock decay between up and lower funnels. Right. So like it's kind of related to what I was trying to ask. I would assume that upper funnel decays slower as opposed to lower funnels where you would see incremental sales quicker. So like do you see the general pattern based on your experience?

Yeah, certainly. Oh, sorry. Lower funnel channels typically have very fast effects. Right. And they almost have to by definition because they are induced by upper funnel spend. So that means they're happening later in the consideration process closer to the point of purchase. And so just there's less things to happen between that spend happening and sales. And so they're typically going to be very close to the purchase. These. Think about what channel we're talking about. We're talking about branded search. So they're searching for the brand, you know, they're maybe going to buy that day. You know, they get the branded search served to them and click on that. The other would be affiliate based commission, which you know, is literally tied mechanically to the purchase itself. And so the, there can be really no time shift there. So I would say almost all lower funnel channels have very short windows, very short effects. Upper funnel channels though can have a wide range. Right? And this is where it can really Dramatically change. Something like Facebook prospecting could have an effect that's like inside of a week or inside of 90 days, depending on the, on the product. Right. A less expensive product, more of an impulse buy, might have a very fast time shift. More of a highly researched or durable product might have a very long window. And so that sort of depends. And then even across category, that's across categories. But then across channels, TV is going to have a longer time shift than a digital channel. Direct mail can have long time shifts because you're talking about it going through the mail, which takes time to even be delivered and then take time to have an effect. So I would say that, you know, lower funnel channels tend to have a very short effect windows, upper funnel channels don't tend to have anything. Right. It can be really, it can be very wide range. That's how I would think about it.

Yeah, yeah, it sounds completely intuitive. The question, sorry, I promised this one last question. How, what is the variability by the brand of what you just described?

Right.

So does it vary by the brand or it's like variability across brands is low.

Yeah. We see a huge amount of variation within brands and across brands. And a lot of it, my view is that actually a lot of it comes down to how the channel, the details of how the channel in particular is managed. Even things like the creative. Right. You could have creative that is very upper funnel. Right. Maybe doesn't even show the product and that might have, you know, in theory a longer time shift and adopt. I honestly think it doesn't work very well generally, but whatever. Or you could have very like call now, go to the website, $5, mention TV and like that is going to have a much shorter time shift. The way, so it's the way that you manage the channel, the creative, all of those things can really impact the time shift. And so you would see, we often do see large amounts of variability within brands, but even, you know, within different channels as they think about, okay, we're using, we're doing Facebook prospecting, but there's really two different campaigns here that are managed very differently and the time shift between those could actually be very different. So that is. And then obviously what Tom was saying is that like, you know, durable goods will on average have longer time shifts and fast moving consumer goods will on average have shorter time shifts. But there's a huge amount of heterogeneity in what we have seen in our experience.

Thank you. It's very useful. Thank you guys. For sure.

Tom was saying is that like, you know, durable goods will on average have longer time shifts and fast moving consumer goods will on average have shorter time shifts. But there's a huge amount of heterogeneity in what we have seen in our experience.

Thank you. It's very useful. Thank you guys. For sure.

And that's another place where we think it can be really risky to use too informative of priors or to say like, oh, you know, we have experience and we know that TV always has an ad stock of 0.7. It just, I don't think that that makes very much sense because TV effectiveness varies so much across brands, for sure, but even within a brand over time, depending on how they're actually managing that channel.

And so being able to say like.

Oh, we're industry experts, you know, it should be X number, we're just going to assume that's true for everyone, I think is, is really bad. And this is why we recommend, you know, when you're configuring a model like this, you want to use broadly uninformative priors and have the model fit the data as best as possible for different parameters as you possibly can to avoid having to make those sorts of assumptions and assuming things that might not actually be applicable to that brand and that particular channel that's, that's trying to do the modeling.

Cool.

We've got about five minutes left. Any, I think maybe time for one or two more questions? All right. If no one's got anything else, I could use a quick break to scarf what's left of my granola bar before my next call. So appreciate everyone making the time to join. This is really fun. If you all have follow up questions, send us emails. We love talking about this stuff. Yeah, really appreciate the engagement. Hopefully this was helpful to you all and we'll see you at the next one. All right, thanks, y'.

All. Take care, y'. All. Bye bye.