Recast validates media-mix-model forecasts like hedge funds backtest: they snapshot each model and revisit it at 7, 30, 60, and 90 days to see what it would have predicted. They use the Continuous Ranked Probability Score to measure accuracy, which rewards both how close actuals land in the forecast distribution and how tight that distribution is. Recast reports 94.7% average accuracy across clients.
Chapters
Recast's media mix models forecast with 94.7% accuracy across our client base. Here is the approach to model validation that helps us get there. Recast approaches model validation like a hedge fund approaches backtesting their trading strategies. The first thing we do is we snapshot every single model that we deploy to our customers. And as we get new data, 7 days later, 30 days later, 60, 90 days later, we go back to that model that's been deployed and we ask it, hey, what would you have forecasted had you known that the customer was going to spend this into those marketing channels? And then we see how well it does. And quite often it does very well. 94.7 is the average. We're also obsessed with making sure that we don't allow any information leakage into these forecasts. A hedge fund including the closing day's price into their model would do really well predicting the opening price, but that would be cheating because that would be feeding the model information that it would not have had. In MMM, there are very close analogs. Things like branded search spend or affiliate commission spend are not things that you could reasonably know in advance. Branded search spend and affiliate commission are outputs of your marketing program, not inputs. And so including them would be cheating. We don't do that. We don't allow any information leakage into our backtests. Finally, the primary metric that we use to model our forecast accuracy, CRPS, the cumulative rank probability score, rewards us both for getting the forecast right, meaning that the actuals come near the middle of our forecast distribution, and for narrowing the distribution. It is the single best way to measure the accuracy of a probabilistic forecast. That we know about. No one takes forecasting as seriously as Recast. A lot of vendors will show you cherry-picked examples where they'll do one backtest if you ask for it. We think a better way is to make it a systematic part of your business and to make it something you do day in, day out, as rigorously as possible.
