In this segment of the video, Michael Kaminsky and a co‑speaker discuss the fundamental challenge of defining the scope of a marketing‑mix‑model (MMM). They emphasize that a model must deliberately omit certain real‑world details, and the key is knowing where to draw that line. The conversation then shifts to how the industry evaluates MMM accuracy, critiquing the over‑reliance on generic error metrics such as MAPE, SMAPE, and WMAPE without clear guidance on whether they are measured in‑sample or out‑of‑sample. The speakers highlight the lack of authoritative benchmarks—no one can confidently state what constitutes a “good” 28‑day out‑of‑sample WMAPE for e‑commerce—illustrated by an anecdote where a claim of 20% MAPE being acceptable is challenged. The overall takeaway is that MMM practitioners need to be explicit about model assumptions, choose appropriate evaluation methods, and avoid relying on vague industry‑wide accuracy standards.
We get into this problem where at some point we have to draw the line and say the model can't represent all of the details of reality. That's how you define a model to some extent. Where do you draw the line? How do you know if you've gotten to a point where we've made the right assumptions and we've cut the right corners in terms of not having to represent all of reality in this model?
Let's pretend you did some of that analysis and you said, it just looks like I need interaction terms and time-varying parameters. Those are fairly common decisions. Now, how accurate is my model? That's a place where we as an industry have been really bad as well. We talk about MAPE, basically prediction error, but we don't talk as much over what is the evaluation criteria. Is this an in-sample test or is this an out-of-sample test? Is it MAPE or is it SMAPE or is it WMAPE? And what is the best we can do? You can't go out to ChatGPT and say, give me an authoritative reference for what a good 28-day out-of-sample WMAPE in e-commerce is for an MMF. Nobody in the entire industry is willing to say, hi, I have an answer to that question and here's the bet I would place, right? I was on a call a couple months ago where somebody said, 20% MAPE is pretty good. And I was like, exactly. I did that face.
I was like, no, it's not.
What is a reasonable number? The answer is twofold. One is, what are you going to do with it? You shouldn't have a budget optimizer that says, if some divine being were to reallocate your spend right now for all eternity, here's the numbers and here's the proportions to what channel. That's not the right way to do it. You're going to increase spend in some channels. You're going to decrease spend in some other channels. You're probably going to go outside of what is reasonably generalizable. A company, every time they told the MMM to optimize, it would recommend spending money in the high-risk channels because those were also the high-reward channels, but those were also the channels where they had the least data. A naive budget optimizer looks at it, takes the point estimate of the value of the channel and says, pump more there. And the answer is no, just no. Create a spend flight with a weekly spend plan change. Take into account the fact that some of these are strategic spends and aren't gonna change. Maybe you have a budget optimizer target and you work towards it and you gain confidence along the way. But you're also, because you're now saying, I'm going to do weekly spend plan changes and I'm going to have weekly variations in spend heading towards the optimizer. I also get weekly checkpoints. Was my model correct or close to correct? I get the chance to sort of naturally without doing an all or nothing geo test, nonetheless vary spend in ways that tell me incrementality. And it sort of naturally becomes part of the forecaster as opposed to being a separate product or a separate set of work.
