Jimmy KimCEO & Founder, Commerce Roundtable · Marketer of 17+ Years, 4x Founder. Former DTC/Retailer & SaaS Founder. Newsletter. Host of ASOM & Send it! Podcast. DTC Event: Commerce Roundtable

How to A/B Test Email Campaigns with Small Lists

19 Jun 2025 · 1 min

Watch on YouTube

0:38

In this segment, the host outlines practical guidance for running A/B tests on email campaigns when the subscriber list is small. He recommends targeting roughly 90% statistical confidence and using rolling tests—sending multiple variations over time—to build a sufficient sample size. Unsubscribe rates are highlighted as a moral compass, signaling potential issues even if click metrics appear strong. The speaker advises watching for spikes in metrics and pulling back when necessary, emphasizing that smaller datasets require longer observation periods (3–6 months) to gather reliable insights. The overall message is to be mindful of data size, quality signals, and timing while continuously refining email tests.

Chapters

aim for kind of 90% confidence. So if your list is tiny, consider doing kind of rolling tests, collecting multiple sends until you reach like some kind of sample size that you feel like has validity. And then also using unsubscribes, right, as kind of a moral compass. So if you see a lot of unsubscribes, if you see kind of spikes in certain things, you know, back off a little bit, even if clicks look good. So just be mindful of what's happening, whether your dataset is large or small. Obviously, the larger your dataset, the quicker you're able to get this feedback loop. So for smaller people, start looking for, I guess, some indication of what's happening before You're 3 or 6 months in running the same test. Subscribe to the pod.