Michael Kaminsky explains why throwing money at geo tests backfires: oversaturation and diminishing returns muddy your results. Skip the 50/50 split. Use synthetic control, matched markets like Etsy does, or stratified sampling to get cleaner signal for less spend.
How do you think about designing experiments that give us at least some amount of signal in this world where everything is hard for all of the reasons that you just mentioned?
I think you're not going to want to hear this, but be ready to spend lots of money. So geo-testing is something that in a sense you can throw money at the problem and make it go away. But finance typically doesn't like that. I say that kind of tongue in cheek, but for those small channels, you can run them as a heavy up test where you spend more into that channel, but the issue— you will get a read from that, but the issue with it is you start running into, I mean, more confounding factors. Like you might be oversaturating the market there, or you might just have diminishing marginal returns as you spend up. So you can get a read by spending more money. You can increase the power of the test by potentially cutting back on spending geos as well. But I mean, again, I guess it's a lot of the times these are decisions that finance doesn't really like. But I guess my overall advice would be if you're using a 50/50 random sample split, don't do it anymore. There are better options out there. I would recommend synthetic control or matched markets. We typically use matched markets at Etsy, but we've also experimented with things like stratified sampling or matched pairs with like a trimming process.
The Continuous Testing Loop That Separates Great Marketing Teams From Average Ones
