Jimmy Kim argues most email A/B tests lack statistical significance because marketers test too small and compare variants that are too similar. Testing a subject line on 10,000 recipients often produces results within the margin of error. Instead, run bigger strategy tests: discount-led versus scarcity-led promotions, story-driven versus product-driven campaigns, or different send frequencies.
Here's the thing. Most A/B tests that people are doing on email don't really have any statistical significance. The brands don't have enough volume. The test variants are too similar and the win is within the margin of error ultimately.
Yeah, I don't have anything to rebuttal there.
Yeah, because most marketers love A/B testing because they think it feels scientific. It feels like a rigor. So they test subject line A versus B to a 10,000-person list and they see a 12% open rate and they're like, that's the winner right there.
Yeah, it's hard to call that a winner, right? It's more just noise. Yeah.
And the whole house of cards is built on tests that didn't actually prove anything.
Nor does actually anyone look at what they're testing and do anything from it anyway. So what's the alternative?
Yeah, so the alternative actually is to test bigger, test things that actually matter. Don't test subject line A, 25% off, or subject line B, save 25%. Test things that are like discount-led promos versus like scarcity-led promos, strategy-led testing, story-led campaigns versus like a product-led campaign. These are real strategic things that actually matter. Yeah, and I think that's the thing. Testings, right? And then you test send once versus send twice in the same week with the same message, for example, right? These are like actual things that you're doing.
