A 3% lift from an incrementality test might sound good but tells you nothing about profitability. Focus on ROI, CPA, and spend instead. A reported 3x ROI could actually range from 0x to 6x, so check the uncertainty interval before you act.
Let's say your most recent incrementality test showed a statistically significant 3% lift. Congratulations! Here's the problem. That doesn't mean anything, and we'll show you why. Whenever you get the results of a lift test, there are two things to be aware of. First of all, we don't care if the channel has any lift. We care if it has a significant amount of lift. And I don't mean statistically significant, I mean actually profitable. A 3% lift doesn't mean anything unless you know how much spend there was and what the baseline of that lift is. We care about things like ROI and CPA. Percent lift on its own doesn't mean anything. The second thing that you have to care about is the uncertainty interval of the test. I see way too many people ignoring this factor. We had a customer come to us celebrating that they got really positive results on one of their channels showing a 3x ROI. Outperforming most of the channels in their mix. But when we dug into the results of the test, the uncertainty intervals showed that the channel could have been anywhere between a 0x ROI and a 6x ROI. That is a massive range, and that's not actually that helpful for decision-making. So the next time you get the results of a lift test, the first thing I want you to know is: what is the actual midpoint of the lift test in the terms that you care about—percent lift doesn't matter—and what is the uncertainty interval of the test?
The Continuous Testing Loop That Separates Great Marketing Teams From Average Ones
