Media Mix Modeling can't reliably measure creative-level ROI because the effect of a single creative change usually gets lost in the noise of aggregate sales data. When vendors claim precise creative performance like "3.9× ROI," they're likely hiding strong assumptions that aren't justified by the data.
Chapters
If your MMM vendor tells you they can estimate performance at the creative level, you should be very skeptical. And here's why. MMMs are top-down models that work on aggregated outcomes, things like total sales or total conversions. They don't observe directly what's happening at the channel level. If you're going to measure the effect of a creative on total sales or total conversions, that creative change has to drive more variation in your sales than the natural noise in the system. And that bar is a lot higher than most people think. So when your vendor comes to you and sells you that creative A has a 3.9x ROI and creative B is a 1.4x ROI, you should do some digging into the assumptions that they're making in order to produce those results, because it's very unlikely that the data is telling them that. Some vendors do things like divide the channel's effect proportionally by spend, Or they fill in some pretty strong assumptions about how creative is having an effect. And that is fine so long as they're telling you that's what they're doing. But if they're telling you that with high precision, their model can make this estimate purely from the data itself, I would be very skeptical.
