This is a YouTube video titled "Active your MMM: Optimize budget with dynamic spend & forecast checks | Modern Analytics Ep. 7" published in April 2024. It features a single speaker, Michael Kaminski, who is an econometrician and marketing science researcher. In this final episode of a series on validating marketing mix models, he explains practical methods for marketers to validate and optimize their marketing budgets using model-driven forecasts and real-world testing.
If a tree falls in the forest and no one is around to hear it, does it really make a sound? If someone runs a marketing mix model but no one actually uses the results, is it really a marketing mixed model at all? I'm Michael Kaminski, econometrician, entrepreneur and marketing science researcher. Welcome to Modern analytics for marketers, our YouTube series that helps marketers learn and put into practice modern marketing measurement frameworks. This is the seventh and last episode in our series on how to validate marketing mix models. Today we're going to talk about about one of the best ways to validate a marketing mix actually using it. The idea is straightforward. If you start using your marketing mix model to make marketing budget allocation decisions and your overall return on marketing investment starts going up, then you know that the recommendations from the MMM were good and likewise the reverse is true. If you start following the recommendations from the MMM and your revenue starts to tank, then it probably means that the marketing mix model isn't working. We think that in general it's important to take a test and learn attitude towards your MMM project. Following the recommendations and seeing what happens is a great way to get started. Now, the approach I just described is a bit informal and doesn't necessarily help us validate specific strategic decisions. In general, your marketing mix model should be able to make falsifiable predictions about what will happen if you make changes to your marketing budget. After all, the whole point of doing a marketing mix modeling project is to determine the best way to optimize your marketing budget and then actually drive business change. We should be able to take the implications of those recommendations and use them to make a series of forecasts about what we expect will happen under different budget scenarios. Then, to actually validate the model, we can compare what actually happens with what the model predicted. Let's talk through an example. Imagine that our marketing mix model indicates that our average channel performance is a 5x ROI, but that our branded search channel has only a 0.8x ROI. This indicates that branded search is dramatically underperforming as a channel and that it's not profitable for us. Now let's imagine a test we can run to validate the model's read on branded search. We can develop a planned budget that reduces branded search spend to zero for some time period, say two weeks, while all other marketing channels continue on business as usual. Then we can make a forecast using that budget which will show our projected revenues over the time of the test, accounting for the reduction in branded search spend. Since branded search spend isn't estimated to be very effective. This forecast will look very similar to a forecast with branded search spend kept on. Then we can just follow the plan. We turn off branded search spend for a couple of weeks and see what happens to revenue. If revenue comes in in line with the predictions or even above it, then the MMM has been validated. We turned off branded search spend. Revenue didn't drop, didn't change at all, indicating that the spend was not in fact incremental. That's the basic idea. There's a lot of complexity that starts to crop up once you account for uncertainty and statistical significance. But our initial recommendation is don't overthink it too much. You should be able to take this idea and use it to get directional reads on whether or not these sort of spend manipulations are generally consistent with the read that you're getting from your marketing mix model. So the overall strategy can be summed up as follows. If a channel is an underperformer, we should reduce spend in that channel next month. If a channel is a high performer, we should increase spend in that channel next month. Then we just compare the actual revenue or customers acquired with holdout predictions from prior months to determine if the actuals are consistent with the forecasts. This consistency with holdout predictions helps us to validate the model. The great part about this strategy is that it helps you to validate the model while also improving your business. You should be iterating towards more and more efficient budgets while also consistently and continuously validating the inferences from your marketing mix model. So we've now reached the end of this initial course on how to validate marketing mix models. If you found this series useful, please like this video so other people can find it as well. Also, please subscribe to our channel. We've got some great plans for more courses on advanced marketing measurement and marketing science methods that you definitely do not want to miss. In the meantime, good luck modeling, good luck running experiments, and we will see you in the next series. Sam.
