Michael KaminskyMarketing Science | Causal Inference | Incrementality

Why 50–50 random geo splits can backfire

Name: Why 50–50 random geo splits can backfire
Uploaded: 2025-10-10T12:00:18+00:00
Duration: 48 s
Channel: Michael Kaminsky
Description: A 50/50 random split of geographic markets can skew your results if a few cities dominate revenue. On Etsy, New York, Los Angeles, Dallas, and Chicago…

10 Oct 2025 · 1 min

Watch on YouTube

0:48

A 50/50 random split of geographic markets can skew your results if a few cities dominate revenue. On Etsy, New York, Los Angeles, Dallas, and Chicago drive outsized sales. If random chance puts most of them in one group, you'll see inflated effect sizes that won't hold when you scale. Use stratified sampling instead.

Chapters

50/50 random sampling. What is the appeal of that? So 50/50 random sampling, you take your pool of geos, you split them randomly into half get the treatment, half get the control. The issue here is that you can really, really easily skew your sample size. So let's just use the United States as an example. At Etsy, some of the major markets disproportionately drive quite a bit of our revenue. So New York, LA, I believe Dallas and Chicago are another big ones. If you happen to randomly sample and you have most of these markets in your control or your treatment group, small changes in those markets are going to cause really big effect sizes, which potentially don't generalize well to the rest of the market once you roll that test out. So you just have to be really, really careful that your random sample in the small sample size is actually representative of everything.