Jimmy Kim and Speaker A explain why most email A/B tests fail: lists under 100,000 to 200,000 contacts lack the volume to detect real differences from minor tweaks. Concept-level tests with bigger changes work better for smaller senders. Large brands can use predictive platforms like Omnisend to find wins in small variations.
You have to test concepts, not just the same variation of the same thing.
You nailed it because concept level tests are big enough to detect actual differences with the volume most brands have. I'm talking about most brands that have sub-100,000, 200,000 lists here, by the way. Like most brands fall into this thing and variation level tests, they don't really tell you anything.
I definitely agree with you. One kind of slight pushback. There are brands at scale where variation level A/B testing actually does produce real wins. So for example, like Omnisend or whoever, predictive testing tools that can do this for you, that's great because because they're aggregating across tons of data and millions of sends.
Again, you have to have huge volume and you're using a built-in tool that aggregates. That's fine. But the average brand, they're just running these manual A/B tests on every send. And honestly, you should stop because you're wasting time and you're not really learning much. It just feels like you're doing stuff.
