Michael Kaminsky explains why p-values mislead marketers: a p-value of 0.03 means there's a 3% chance you'd see this result if the campaign did nothing, not that it actually worked. Statistical significance just proves something happened, not that the lift justifies the spend. Define the minimum lift you need upfront, then check if reality hits that number.
Chapters
P-values are at the center of one of the most expensive mistakes in marketing analytics. Let's say you run a marketing incrementality experiment. The results come back and you get a p-value of 0.03. It's statistically significant. Your team celebrates, banks the win, and you hear someone say, "So there's a 97% chance the experiment worked." And it sort of makes sense, right? But that's not actually what a p-value means. Here's what it's actually telling you. A business's sales data move around every single week randomly because of the weather, because of competitors, day of week effects, and other random factors that we'll never be able to fully explain. That movement happens whether or not the marketing campaigns are actually driving true incremental effects. The p-value is asking a very specific question. If a given campaign had zero effect on behavior, none, nothing, the ads might as well not have run, what are the odds that you'd still see a result this big just from that normal background noise? P 0.03 means about 3%. 3% of the time, even when this campaign had no real incremental effect on sales, we'd still see results this large from an experiment, just from that random background noise. But back to our marketing experiment. This means that our test actually answered one question: How likely is this result if our campaign truly did nothing? But the problem is that people hear a different question: How likely is it that the campaign worked? They sound similar, but they're actually very different questions, and it's important to be precise about them. There's even a deeper problem here. A standard p-value in a null hypothesis significant test only compares your result to one benchmark: zero effect. Not a small effect, not a break-even effect, but truly zero. So statistically significant means one thing: the campaign probably did something. But for a budget decision, something isn't enough. You need to know if the effect was large enough to actually be worth what you spent. A campaign can clear the p-value threshold legitimately correctly and still deliver a lift so small it doesn't come close to justifying the So even when p-values are working exactly as intended, they're often answering a different question than the one that really matters. The test asks, did this do anything? The business question should be, did this do enough? So what's the right focus after an experiment? Don't just ask yourself, was this significant? But was the effect large enough to matter? And before you run the experiment, you should define what meaningful lift looks like for your business. What effect size would actually justify increasing ad spend? A p-value can tell you if the campaign did something, but only you can decide if that something was worth it.
