Key features and limitations | Google Meridian MMM

Foreign hey, Michael from Recast here wanted to record this quick video to talk a little bit about the new open source MMM package that Google just released called Meridian. I think there's a lot to like about this package. It's going to be great for people looking to experiment with mmm. There's some really cool new key features but also some limitations that I want to chat through that probably practitioners should be aware of. So first, what's the like? First of all, it's fully Bayesian. I am not a zealot, but do really think that the Bayesian approach is the right approach for these complicated observational models like media mix models. So it's Bayesian from the ground up that makes the assumptions that it's making really clear. It means that uncertainty propagates through the package correctly. So a lot to like about that. Second, it's a geographical based hierarchical model which is just like lightweight mmm, which is the sort of old open source sort of maintained package from Google. There's a lot of good reasons to like this approach. It allows for more variation in the data. It allows you to exploit that variation I think in smart ways. So overall makes a lot of sense to me. Some of the new things that weren't in lightweight MMM is they're introducing a time varying intercept or baseline sales. The documentation on this is a little unclear. They talk about that you'll be able to specify how this time varying intercept can change with the use of knots, which is going to be a hyper parameter. But overall I think an important step forward. They introduced two new features around incorporating other marketing data that isn't just like spend or impressions. So one is the ability to incorporate search volume. Some researchers at Google had published a paper indicating there might be some ways to address the endogeneity of search marketing activity by accounting for the amount of search volume that's actually happening for a brand or a product class. And so it looks like this package is going to have a way of incorporating that hopefully in a smart way. A lot of the details are still a little bit unclear to me. The second is the ability to incorporate reach and frequency measures. This is a thing that I think is really interesting and exciting but also honestly pretty fraught. Reach and frequency measures are used a lot in like video advertising, especially linear tv. That's how buyers think about buying media is trying to hit some reach and frequency targets. The problem is that those measures are often based on surveys. Their measurement is opaque. It can change over time. Different vendors will measure reach and frequency differently things to be careful of, but really cool to see this coming into a package potentially could be useful. And the last thing that I really like about Meridian is there's just lots of great documentation and discussion about doing causal inference in mmm. There's a lot of discussion of the challenges and the assumptions that you have to make and what the limitations are. And I think that Google putting their weight behind a fully Bayesian model I think again adds a lot of credibility to taking this approach. And also their discussion of the limitations I think is really helpful because if you're doing this sort of work, you need to be really aware of the limitations, what's possible, what's not, what assumptions need to be made in order to be able to do this sort of work. So that's all the great stuff. I want to talk a little bit about some of the limitations. First is that there's no time varying covariates. Like a lot of the open source packages that are available today, Meridian makes the assumption that marketing performance within a given channel is fixed over the entire time period being analyzed. So if you're analyzing the last two and a half years worth of data, you're going to get a result back that says, hey, Facebook has an ROI of 3.5x over the last two and a half years and it's just one number, right? It doesn't change, it doesn't get worse, it doesn't get better. That doesn't match with what I think most marketers and most marketing scientists believe about how marketing works in the real world. Marketing performance within a channel can change fairly dramatically over fairly short time periods. It could be due to seasonality, it could be due to creative changes, it could be due to the platform itself. As I'm recording this, there's a lot of people talking about seeing degraded performance in Facebook. And so if your marketing mix model or media mix model isn't able to capture the way that marketing channel performance actually changes over time, the model is effectively going to be misspecified and it can lead to really bad results. Meridian talks about their new ability to incorporate experimental results into the model via priors and I think on the one hand what they have done is nice, which is give users more ability to more intuitively set priors based on ROI instead of based on a beta parameter. But that being said, they haven't actually solved the problem of how to incorporate experimental results because they don't have time varying covariates. If you have multiple different experimental results taken at different points in time, it's unclear to how to how to actually incorporate that information correctly. So most of the brands that I work with at Recast, they run experiments a lot. They run experiments once a quarter in their big channels, and the results of those experiments are often different. Right? If you have a seasonal business, you run an experiment in July, you run an experiment in December, you're going to get different results. And so the Meridian package doesn't really provide any guidance as to how to pull that story together when you're running multiple different experiments over the time period that's being analyzed in the mmm. Another limitation is that they don't really have explicit handling of upper funnel versus lower funnel channels and the interaction between them. So there's no explicit modeling of the relationship between TV ads and then search volume on the other side. I think that that's a really, really important limitation in, in order to truly understand the effect of video advertising, you need to understand how it drives search volume. And that doesn't appear to be actually handled in this library. Another limitation is they don't seem to have a lot of tooling around constrained optimization. So they have an optimization tool in the light in the package itself, but the optimization tool is limited, right? It only really lets you say, hey, we have a budget. What's the optimal spend allocation if we use up all of that budget? But practically, when marketers are actually building budgets, they face a lot of constraints. They may have locked in spend for some channel. There may be organizational constraints about how what's the most or the minimum amount that you can spend in a given channel. And so you really need a constrained optimization platform in order to be able to help marketers actually do their job. This idea of like, the theoretically optimal budget is cool, right? And I'd be interesting to see, but it's not actually practically helpful given the constraints that marketers really face. And the last limitation that I wanted to call out is just that there's limited features around model validation and stability evaluation. One thing that we like to say at Recast is that running an MMM is really easy, trivially easy. Proving that you have the right MIM is really, really difficult. And so when you're actually doing this modeling process, there's a lot of steps that go into building a good model that require a lot of engineering work in order to make happen. Doing parameter recovery, doing stability and robustness checks, doing repeated holdout forecast accuracy checks, all of those things are a lot of work, but they don't seem to be actually built into the package itself. So overall, a lot to be excited about. A really great open source package. I'm really excited to see how people start to use it. Always excited to see developments in the space. Drop a note in the comments if you started to use Meridian, what you've seen, what you like, what you have questions about.