Robyn creator: Understanding MMM & Project Robyn | Gufeng Zhou (Meta) | Coffee Breaks

Foreign.

I am really, really excited for the conversation today. So I'm Michael, I'm one of the co founders, co CEOs of Recast. Welcome everyone to the Marketing Measurement coffee break. We are thrilled to be welcoming Gufeng Zhao to be here with us today. Gufeng is on the marketing science team at Meta. Many of y' all know who he is. He is the creator, DS lead and PM of Project Robin, the open source MMM tool. We're excited to have him. Today we're going to have a really deep conversation about some of the choices that goofing has and trade offs that goofing has had to make. As he's thought about Project Robin and an open source MMM tool. We're going to go deep on different modeling philosophies, different ways to approach this problem and as always we will not be doing any pitching so we're going to keep this really focused on the content. If you have questions please drop them into the chat. Myself and my team will be keeping an eye on the chat. As good questions come up we'll sort of feed them in so please definitely drop them there. So you Fang, welcome. Excited to have you. Without further ado, I want to get to my first question which is MMM models, media mix modeling in general, they tend to be really really powerful. Right. They're very flexible modeling frameworks which experienced modelers know means that they're also very easy to abuse. Right. If you have enough parameters you can overfit to anything in your experience. What are some of the common mistakes that you see people making when they're using Robin or these other really flexible frameworks? I would love to just hear your opinion on that.

Thanks for the question. So the common pitfalls I would say this. With the age of slightly more automated MMMS models the danger is actually even greater to misuse this kind of model because when you had a team of modeler back then you be of course more cautious about oh are we having the right data set? We assess is there any measurement issue in terms of the numbers are right and then you put a lot of energy into feature selection and then put a lot of energy into of course dealing with multicollinearity and all that kind of stuff. Ensure that how about all the regression there? You put a lot of energy there. What is the trend? How do I calculate for calculate seasonality and then now everything's this kind of stuff is lot automated now and it makes it easier to misuse this kind of tool actually.

Right.

You don't really think about oh what is actually, let's say the right structure for my data set.

Right.

Do I need to consider maybe upper, lower funnel mean in my modeling? This is a lot of this might, you know, might be just kind of fading because the easy, the user friendliness of these tools. So I would say this, right, that it's more of the SaaS mindset that people goes in and goes out.

Right.

Which is great. You know, democratizing the technique is our purpose as well. But at the same time, right. We know that the industry, especially from the more academical side is, you know, pledging for more cautious, more, you know, domain knowledge, more of the statistical knowledge into this and we actually agree with it.

So I think that makes a lot of sense and sort of summarize, you know, this is a pattern that we've seen and even, you know, I myself have fallen into like you write the script to do the thing and it does the thing automatically for you and you forget to check all of the inputs. And I think, you know, in some of the slack chats that I'm in talking about, mmm, you definitely get this sense that there are people who are trying to just like take a data set, feed it into the algorithm and get something on the other side. But that, you know, doesn't always work, especially in domains that are as complex as media mix modeling. So practically, I guess, what advice would you give to someone who's like, I don't know, a senior analyst, data scientist, they have some skills but maybe don't have a lot of background in media mix modeling. Like what advice would you give them? What are the practical things that they can do to avoid falling into some of these traps?

Yes, well, one thing top of mind would be that talk to your media team like this is what I, what we've seen that many data scientists, they might be, you know, equipped with a lot of skills but not necessarily a lot of, you know, knowledge or hands on, on the ground experience and then, and probably don't have the structure internally to naturally approach these kind of media teams. What happens is that they're missing a lot of context for setting up Right. Models.

Right.

For example, the context of, oh, maybe a channel, certain channel is being over invested because they know that they have a lot of frequency there. There are a lot of this kind of inherent knowledge that you probably won't get out just by getting a CSV file.

Right.

So this is quite important for modelers or you know, tool users, right. To talk to your media team. They are your most important stakeholder in a sense. And get as much domain knowledge out as possible and this will compensate a lot for what's missing.

Right.

And then of course, you know, if possible, talk to an experienced modeler as well, what kind of other things that he would look into. And then it's. Yeah, in this context, then basically guarantee a more comprehensive, but also like guarantee the most out of the, let's say, the causal structure in a sense.

Right.

So, so yes, that's very important.

I think that is such great advice. And you said it so simply, like talk to the media team, talk to the marketers. We at Recast, we hire a bunch of data scientists from like, you know, academia.

Right.

They like, you know, have worked doing very advanced modeling, but don't necessarily have a background in marketing. And often when they start at Recast, they have a very simplified idea of how marketing works and they're like, oh yeah, you just like put dollars in and then ads go out and then people convert. But it's like all of these different marketing channels are so much more complex than that. The way you can do different bidding on different platforms, the way media is bought in different ways. Like TV can be bought at Upfront or it can be bought at Remnant and that's really different strategically. And how it's price and like there's so much detail here and you're totally right that as a modeler you really need to understand that detail and how it works for your business and your context in order to be able to, you know, operate these models correctly. Changing topics a little bit. Most of the other open source MMM tools that have been gaining in popularity, whether that's lightweight MMM or the one that the PMC Labs team have been developing, they're Bayesian indeed. But Robin isn't. So Robin sort of stands out in being not a Bayesian based model. I'd love to hear from you. How do you think about the trade offs of those different approaches and sort of how did you land on the approach that you all ended up taking at Robin?

Yes. So we're actually not very much preferring in general preferring or not preferring Bayesian models. So the comparison when you talk about Bayesian and the other side is always frequentist.

Right.

It's like there's a, you know, fight or, you know, eternal fight between these two school of actually philosophies.

Right.

It's not just statistics. So in this sense we've made this decision out of very concrete reasons that maybe very few of you guys might know that there was a Robin 1.0 version that we beta tested. With very limited users. Actually alpha tested that it was not looking closely like what it is today. So we were not having the multi objective functions. So we were basically optimizing for just for the smallest prediction error, so to say.

Right.

And then found out that oh, the results were of course often not interpretable.

Right.

And then we're starting to think that how could we improve this point? Because especially as a manual model, you would know that you would tweak certain things, you make certain decisions so that you get a halfway, okay, model out. By doing that you're actually having a lot of inherent criteria and they could be borrowed. Other users, other modelers, right. So there are this common ground. So we try to extract that and basically realize that, okay, there's actually a business side of goals, objectives that we generally do not account for when we model.

Right.

So the result needs to make sense. So the idea was behind how could we account for that part.

Right.

Make it more closer to the business reality in this sense. And this is of course a very big task.

Right.

But we did borrow it from the optimization field. Optimization, right. Which is you know, I guess officially part of the AI. But in the end it's part of, you know, it's an optimization problem with multiple objectives. So we're dealing with the real life problem that is just not simply just forecasting what is the most accurate, you know, but we need to explain things, we deal with inference. It's very important, if not more important.

Right.

For mmms. So, so we need to have an objective in the optimization that accounts for that part. This is actually remember Weast had this blog post that was of that author that was saying optimizing for politics, that is that we call it the decomposition distance, the decom rssd. And that accounts for, or let's say this in the practice, it eliminates those very extreme results. And we have seen this works in the practice quite well.

Right.

So of course we also earned a lot of critics that it might bias results. But we will have a different view on what is bias here.

Right.

Because when you're optimizing towards different more than one goals, then the bias needs to actually be redefined as well. It's not just the simple statistical bias anymore. So but altogether we've decided to do multi objective optimization and then that would require or say what would be the combination of an evolutionary algorithm and multiple objectives plus Bayesian with a lot of burn ins, with a lot of simulations beforehand. So it's actually out of the practical reasons we don't want to do Iterations on top of iterations that will explode the runtime a bit.

Right.

It's out of that reason. But of course, for me personally, I do believe that there's a truth of, you know, your performance.

Right.

There's out there somewhere right in the universe is truth of performance. I like to think about things like that in a sense that, you know, capturing that truth. Basically the concept of confidence interval, you know, for me, just personally.

Right.

It kind of explains this process a bit better. Then of course credible interval is easier to interpret, of course.

Right.

In a sense, but it explains things in uncertainty. And this is very philosophical though, but our main reason was indeed that we do not want to explode computation time.

Computation can definitely be really tricky, especially in these really large multi dimensional spaces. And I think you're like, there's just trade offs on all sides. You have to choose what you're going to optimize for and choose what's most important for you because you don't have infinite computation. And so you have to figure out like exactly what corners are going to get cut in whatever algorithm you're going to be developing. I want to switch topics again a little bit. A lot of people are talking right now about incrementality testing, whether that's geotesting or, you know, conversion list studies in the Meta platform and how do you like how that can integrate with media mix modeling. I'd love to hear, in your opinion, what is the best practice for using these techniques together, if at all?

Yes. So we know that all the major players in the industry are advocating for use all your powers and do all the techniques. We have the trinity of marketing, that is attribution. We have MMM and we have experimentation.

Right.

So it is right that at this time of changes you want to use all the techniques because they all have strength and they all have their weaknesses. You probably get the best out of it when you are using all. But then of course it comes the question of how do you rectify between them? How do you calibrate between them? So at Meta, we believe that experimentation is the golden standard.

Right.

Of course, you know, academically there is, it's impossible to exactly measure the causal effect.

Right.

It's just academically not impossible. Even, you know, randomization has a random number generator behind and you can argue that is not random. But we're, we're not to talk about those kind of theoretical levels. We're talking about, you know, the best you can get. And that's what gold standard means.

Right.

The best you can get. And Then we do believe that experimentation in marketing is a gold standard.

Right.

So that is the closest you could get to causal effect. And it is the closest, the strongest causal inference method. So we also distinguish the word causal inference from causal effect because sometimes these two aspects get cause mixed up and then you know, a solution claims to be causal. They're doing a little bit of causal inference, but claiming full of the causal effect credit.

Right.

So that would be something that would say overselling a little bit.

Right.

But in general we do believe that experimentation is the gold standard, especially when you're having a very powerful test like meta conversion lift study of course based on know millions of IDs and devices and you know, is of course a very powerful randomization that you know, you usually won't get.

Right.

Geo experiments are very practical.

Right.

Is of course, you know, way less powerful due to the number of observations as you know, the samples that are possible.

Right.

Geographic units. You can't compete that with the millions of samples as IDs and humans and devices. But nevertheless the next closer thing that you can do. And then of course we do believe there's hierarchy of results here. But then again, experiments are limited, bounded for a certain time. You can measure a couple of weeks. That's it. It's a point estimate over that week. That's it.

Right.

And then you want to have of course over longer period. And it's expensive in the sense that, well, sometimes you need to afford bigger holdout. Not every media team is happy to hear that, that oh, I cannot reach 30% of my audience. What are you saying?

Right.

For four weeks.

Right.

Maybe this is before Christmas and that's, you know, gonna be.

Right.

You know, preventing them to hitting their goals. Right. So there's a lot of of course practical difficulties there as well. So we understand that you need other techniques to, to accommodate for that. So when you're doing this and we add, we do support that, but you need to take into account the scope of differences between these techniques. So for example, MMM includes short term effect and at least we, we're refusing to call it long term effect. So we just call it carryover because it's not long term and you have short term effect and carryover effect in MMM included inherently.

Right.

All MMM do have those and experiments conversion lip studies do not.

Right.

So because it's a short period and it does not account for historical carryover into that period. And this is very important context, that means when you're calibrating, you need to know that, well, you know, maybe before the study, you had a big campaign for month.

Right.

And then you have a lot of user buying within your study period. They're because of that campaign before and not because the spend that you have within the study.

Right.

Mm. Is accounting for this part and the experiment not. So there's an inherent difference. You need to account for this. Otherwise, practically, how do.

I mean, I totally recognize the problem. It seems like a really hard problem. Is there. How do you account for that? Can you. Do you have to just do like a rule of thumb or look at the, you know, the carryover effect from the MMM and sort of back into it or like, what's the recommendation practically for how to make that happen?

Well, practically would be just something like if you have a comparable study, first of all, you need to measure the same span more or less as in modeling. Mmm. If that is given, then the easier thing to just look at the results, do a manual comparison. Can you explain the difference? Can it be explained by the missing carryover effect? Right. Do you know, qualitative judgment there? So if this is the case, and technically speaking, you could just, you know, easy way would be just doing a multiplier.

Right.

So If I measure 100 in lift and it's 130 in. Mmm, so it's 1.3.

Right.

That's easy. But then you want to account for it more systematically. Actually, Robin does that, actually. So we separate the modeled decomposition so the contribution into the immediate and carryover part. So it will be only calibrating the immediate part of the effect. So leave out the carryover, for example.

Right.

But this is just one of the many scope differences. It's not impossible to account for all of them, but at least you want to make sure that you account for one or two of these aspects to be a bit more accurate.

So I think that makes a lot of sense. And I, you know, I think you were the author on this blog post that I reference all of the time that like, sort of it has this great illustration of the different carryover effects from the different time periods and how that interacts with like, what exactly what the experiment is measuring and what something like an MMM is measuring. So I'll say everyone should go read this blog post. It's really, really great. And I. But I think the, the overarching point here is that as analysts and scientists, we just always need to be really thoughtful about exactly what are we measuring. Right. And like, it's really to say, like, oh, it's an experiment and it's an mmm and they're the same but they're, they're often not. And there are these very subtle differences in, in measurement and exactly what is being measured that you just have to be thoughtful about and it's, it's easy to get mixed up and get misled. And this is why, you know, we have jobs, I guess is because like this stuff is hard and you need to actually think about it. And I think that that for me is like the main, the main thing to keep reminding myself always.

Yes, totally. So I mean in the end, right, we do believe that, well, the industry has been evolving, you know, since all the priority changes and then these kind of problems now serve us more.

Right.

So it's a process of the industry now basically evolving together. So in this sense I won't, you know, take it too hard on yourself, you know, kind of just say, well, you know, but now we are starting to consider these problems. So it's a good development.

Right.

And we might look into even more scope differences and solve for that as well, account for that and maybe integrate them into our solutions as well so that measurement becomes more comparable between different techniques. And the education of the market takes time. It always takes a lot of time actually. So this is actually a process. So actually we are happy to see that this is happening. That you know, the post got read, you know, by people that is, you know, it's not, you know, they had values and then, you know, it got picked up. We talked to other providers as well and they're like, you know, oh, you know, we are actually doing similar things in the different ways but you know, now put into more, you know, consolidated context and this is that.

Right.

So scope differences of different measurement techniques. So I think it's a good development in general.

Yeah, totally agree. Last question that I wanted to ask. You know we've been, people have been talking about triangulation, you've mentioned it and I think in the context of lift tests it's pretty easy to reason them out because a well run experiment as you mentioned is the gold standard. Right. It gives us sort of a really because of our ability to manipulate the intervention, it makes the causal inference piece a lot easier. Right. Because you don't have to make as many assumptions. The third leg of the triangle which is like multi touch attribution or digital tracking I think is a little bit harder to reason about. I think there's a very real sense in which it does make sense to sort of calibrate with what you're seeing in digital tracking. Like there is signal, there But I often struggle with how much. Right, because there are times when it's going to be really biased and times when maybe it actually is, you know, a really good signal. How do you think about incorporating that information from MTA or in platform metrics or whatever sort of digital tracking into this scientific learning process, whether that's experimentation or. Mmm.

Yes. I mean, attribution does have its values even today I would say this, right? I, I don't agree with those that say, oh, you know, this is just, you know, worthless. It's not true. Because, for example, for certain industries it was and still is a very good proxy to causality to incrementality. For certain industries, like gaming is always the one that I mentioned, but because of the conversion path is pretty short. So the type of conversion they're measuring is quite quick. So app install is the most important one and it's. And this result in that the last click or last touch attribution is a good enough proxy, right? And it's cheap to implement, it's really fast and really granular. So you can't, you know, you can't say no to this kind of solution for your product.

Right.

So and this is actually proven over and over again that for these kind of industries it's a very good proxy. The correlation is very high. So there's no reason to not use. This is the first point. The second point would be that like even for, let's say, well, you know, products with longer purchase cycle and all that, of course is not ideal. But MMM does have a granularity problem, right? We don't get to the granularity campaign level. Even lower creatives, it's almost impossible in MMM to do that.

Right.

And then where do you get otherwise the campaign level or even creative level numbers, they are in some kind of attributional models inside, right. Be it on platform, be it your own solutions or third party solutions, right. And over there you get a sense of what is what.

Right.

And with domain knowledge, again, right, you account for maybe, oh, this is a rather upper funnel channel, so it might be missing a lot of view through conversion. So somehow account for that maybe, but this does compensate the granularity and then by that with extension the actionability for advertisers, right? They sometimes do want to have a number on creatives and then, you know, and it's. Right, rightly so. So they would need any. Some sort of. And this is almost the only source of a number for that. Right. So that's why we need to be more cautious about what kind of numbers you're looking at. But to say that these numbers have no value is not true.

I totally agree. I would never say that those numbers have no value. I think and I think you summed it up really well and I have to talk about it less even like gaming is a really good example where I agree that attribution gives a lot of signal. I also just think that like for smaller companies, right, if you're a small e commerce company and you're only advertising on meta or only advertising on Meta and Google like Last Touch attribution is really good. Like it's like that's like a really good indicator of what's actually driving your business and it's only when you get bigger and more complicated where you need some of these other tools and where the flaws in attribution are start to show. And then I think you're exactly right that especially for like campaign level and creative level analysis, it absolutely like these statistical models are just not going to be able to tell you anything. They're just not going to have enough power. So totally 100% agree with that. And with that we are out of time. So goofing, because he is in Singapore has to go start his day. So you know, he is about to go off to work and some of the other people on the call are about to go to bed. So thank you everyone for making the time to join us. Really, really appreciate it. Thank you so much. Goofing. A couple of quick calls to action for folks if you haven't already. Give us both a follow on LinkedIn. Both of us are always talking about marketing measurement, marketing analytics, causality. Mmm. And I think most of the people on this call have probably already checked out. Robin. Mmm. But if you haven't, there's a ton of great resources on the website, a bunch of interesting documentation, theoretical discussion of some of these really important points that we talked about today. So highly recommend everyone dig into that. Gu Feng, thank you so much. Go off and enjoy your Thursday.

Thank you as well. So yeah, talk to you soon.