Tom Vladik from Recast walks through the platform's holdout testing feature, which validates how well the media mix model predicts marketing performance. The tool lets you set aside historical data to test if the model's forecasts actually match what happened.
Hi, I'm Tom Vladik, co founder of Recast, a media mix modeling platform where the world's most sophisticated brands measure and optimize their marketing spend. So today I want to talk to you about back testing, or holdout testing, and how we've built it directly into the app. Now, before I go to the app, I just wanted to mention that we've done some writing on this topic that summarizes how we do it, some things that we do differently, and why we think they're important. There's a great blog post called how we do holdout testing at Recast, and I highly recommend that you go check it out. So let's go to the app. Now, before I dive into what we do, I just wanted to mention that we think that backtests are one of the most important modeling features in a Mediamix model. The only way that you can know that your model is performing correctly is whether or not it can predict the future. And ideally, not just the future, but any future. Because when you do optimizations, you want to know what will happen if you cut TV spend in half or you double it. And the model needs to be able to accurately predict what will happen in any possible scenario. And so we take this very seriously, and we've built it directly into the app to make it super transparent to our customers how the model is performing. So you're on the demo dashboard now, and you can see we're on the sort of insight section, and all the way to the right, we have the back tests. Now, let me explain how this works. Recast updates every week. That means every week we get a new set of data, and we completely refit the model to that data. What we also do is every time we get a new data set, we go back in time and we see how previously deployed models would have done, forecasting the new data all the way up from when they were trained to current day. So you can see that at the top here we have a table, and each row in this table represents a model that was deployed sometime in the past. Sometime in the past here ranges from 98 days ago to 7 days ago. The 7 days ago model is the most recent model that we deployed before we just got a new set of data. So for each model, we show what actually happened. So between 98 days ago and today, this demo client got $20.4 million in sales in that model, knowing only what you would have spent would have forecasted somewhere between 18.3 and $23.9 million of sales, with an average error of 2.73%. Now, for each model, we have a corresponding statistics for 70 days ago, for 42 days ago, 30 days ago, and so on. We also have plots showing the forecast range against the outcome. And the plots are both daily and cumulative. The daily is the daily prediction against the daily actual result, and the cumulative is a running total over the period. And you can see as I click through these, what some of these look like. But that's not all. If I scroll down here, we also have what we call a reconciliation between the most recent model and the model that was deployed in the past. Now, how we do this is we show how the model has changed in assigning credit. So I'll just pick one row here as an illustrative example. So in Facebook prospecting between 70 days ago and today, you spent or the demo client here spent approximately $1.2 million on that channel. The model 70 days ago would have forecast that $9.3 million worth of total sales would have been caused by that spend. But the current model that has actually seen the sales and refit the model to that data now only estimates that about $8.6 million worth of sales came from that same spend, resulting in a downward revision of approximately $700,000. We do this for every channel and the intercept and spikes, and we can roll it up into a total view of how the model has changed between 70 days ago or 98 or 7, whenever the model was deployed in the past and today. How has the model changed its estimates is what we're trying to answer here. So, as you can see, we take backtests and reconciliation incredibly seriously at Recast. If you would like a media mix modeling partner that does so as well, go to getrecast.com and hit book a demo. Thank you very much.
