Michael Kaminsky explains the winner's curse in A/B testing: when Microsoft reran statistically significant experiments, effect sizes almost always shrank, revealing that original estimates were inflated. The problem stems from focusing only on significant results, which are the most extreme random deviations. Three fixes: use confidence intervals instead of point estimates, include non-significant results, and apply shrinkage techniques.
Chapters
Microsoft runs thousands of A/B tests a year. When they started rerunning some of their winners—those tests that came back statistically significant—they found something uncomfortable. The effects were almost always smaller the second time around. And sometimes a lot smaller. The wins were real, but the numbers were inflated. This isn't just a Microsoft problem. It's a fundamental problem with the concept of statistical significance. It's called the winner's curse, and if you're running A/B tests and only paying attention to the ones that win, your results might be overstated too. The winner's curse is one of the wildest, most most non-intuitive facts that comes up in statistics, but it's extremely important to understand if you're doing any type of A/B testing or experimentation. The problem is that when you only look at the experiments you ran that were statistically significant, the results you're seeing are actually likely to be dramatic overestimates of the true difference between your experimental variants. This problem gets worse if you're working mostly with small sample sizes where the results are often barely significant. Here's the issue: if you run lots of experiments, we expect that the results we see in the experiment will randomly vary around the true difference. So if If we repeat the exact same experiment multiple times, we'll often get different results just due to random variation. But within that random variation, if we select only the results that are statistically significant, we'll see that we are only selecting the experiments that have the most random variation. The effect isn't really this large, but our selection procedure causes us to only focus on the versions of the experiment that are the most wrong. This is the winner's curse. In high-volume experimental settings like online experiments or marketing, we expect that if we filter on statistical significance and look at the point estimates for the experimental results, we will consistently overestimate the effects of our winners. So what can we do about it? First, by focusing on the full uncertainty interval produced by the experiment, we can avoid being fooled by the point estimate, which will be inflated. Second, we shouldn't just throw out experimental results that aren't statistically significant. They still contain important information and give context about what reasonable effect sizes might be for our business and the types of experiments we're running. Third, we should keep in mind that our estimates are likely biased due to the winner's curse and potentially use regularization to downweight those effects.
