A/A Testing: How to Check If Your Experimentation Platform Actually Works

In order for a scale to be useful, it should give the same measurements as other scales. 1 kilogram on my scale should be 1 kilogram on your scale too. Experimentation platforms are similar. They need to be calibrated to make sure that you're measuring what you think you're measuring. Here's what I mean. Imagine a product team ships a new checkout experience with fewer steps than their evergreen flow. They test how performant it is, and when their experiment results come back, Conversion is up and the read is statistically significant. Everybody's happy, but there's a question nobody asked. How do we know that our experiment system is working as intended? Not whether the new checkout flow works, but whether the system for measuring it works. In other words, who's watching the watchman? How do we test the A/B testing machine itself? And that's what an A/A test is for. Not an A/B test, an A/A test. You run an experiment splitting users into two groups, but you show both groups the exact same thing. It's variant A versus variant A instead of variant A versus variant B. This is like weighing the same stone on two scales. You need to make sure they're both outputting the same number. And if they don't, it means that at least one of the scales is wrong. A/A tests are critical in any field that runs experiments. Not just marketing science. They can be used by teams working with product growth, pricing, you name it. One important point here is that while a scale is directly measuring an observed value, your A/B testing software is measuring a distribution around an unobserved value. So we need to take into account the random variability when we're evaluating our tools. With the usual p-value less than or equal to the 5% threshold, we expect that about 1 in 20 of these A/A tests will flag by pure luck a difference between the two experiment groups. This doesn't That doesn't actually mean the A/B testing machinery is broken. It's just how random variability works in statistics. So it's reasonable to shrug off a lone surprising result in an A/A test, but a pattern of them or a difference way too big to be luck alone? That's your system telling you that something is very wrong. Here's an example to hopefully help make this real. Microsoft runs experiments at a massive scale across the many businesses they operate. To do this fast and at scale, they historically assigned users to testing groups based on a hash function that divided these users into buckets. The problem Microsoft eventually ran into, unbeknownst to them, was that they kept reusing the same buckets across experiments without fully reshuffling their users. So when they eventually ran an AA test with two identical groups, they should not have found a significant result, but the results kept coming back significant. It turned out users in these buckets were also in previous experiments run by Microsoft, and they carried residual effects from those earlier experiments into the AA test and into other tests that they were supposed to be evaluating. That's what made the groups look different. They weren't actually fresh new groups, but were contaminated from previous tests. An AA test caught this phenomenon at Microsoft and had a huge impact on their experimentation program. They started reshuffling users between experiments, and the ghost differences Vanished. A/B tests can also be used to catch pipeline errors. Imagine two groups looking at an identical landing page for an e-commerce business, except a tracking pixel fires for one group and not for the other due to a glitch. That tracking pixel is what records user activity, so the group with the proper pixel firing logs extra conversion events, which an experimentation system reads as more users coming in one group. This difference appears despite nothing in the conversion experience actually changing. Except a broken logging system. The AA test flags a difference that shouldn't exist, which is how you find the broken experimentation pipelines. AA testing is also a tool to catch experiment calibration issues. Say in your experiment you randomize the intervention by user but score the test by page view. An edge case here is that one single user might trigger 50 page views in a day. Your stats engine would then think that those are 50 independent events, but they're not. They're all from the same user. And because the engine counts them as independent, it may badly underestimate the uncertainty around your results. The engine thinks it's being precise, but it's actually overconfident, and your 5% false positive rate balloons to 15, 20%, or even worse. You'd never catch that with just one test, but if you can run 100 AAA tests and look at the spread of results, you will see that while significance should show up about 5% of the time, it's actually showing up about 20%. This is evidence of a miscalibrated engine. Bad randomization, broken logging, a miscalibrated stats engine, 3 different failures caught by the same detector, the AA test. The playbook is pretty simple. Run AA tests continuously to build and maintain trust in your experimentation platform itself. Don't run just one AA test, run a batch. Significant should show up in about 1 in every 20 experiments if you're using a 5% threshold. Much more than that, Stop. Something is broken and every readout is going to be suspect until you find and fix the issue. Microsoft didn't run just one AA test and call it a day. Teams operating at that scale run them continuously. So if you've never checked the instrument, your next AB test shouldn't be A versus B, it should be A versus A.