Why the default answer is no
An A/A test is usually asked for as a trust exercise. The problem is that the version people run, twoidentical experiences read on a conversion-rate p-value, is the weakest instrument available forthe job. Three things go wrong with it.
A pass proves almost nothing. A non-significant A/A does not demonstrate the tool is sound, itonly demonstrates the test lacked the power to find a difference. To rule out a difference assmall as 0.1% relative on a 5% baseline conversion rate you would need roughly 298 millionvisitors per arm at 95% confidence and 80% power. AB Tasty put the same figure at around 300million.
A failure is expected. At a 0.05 threshold, one A/A in twenty comes back significant whileeverything is working correctly. Teams that read that as a broken tool then distrust results theyshould have believed. Georgi Georgiev's objection is exactly this: the significance thresholdalready accounts for the outcome, so a rare significant result is a predictable event, notevidence of a fault.
Adding arms makes it worse. A/A/B and A/A/B/B designs are often proposed as a way to getvalidation and a real test in one shot. Every extra arm is another comparison and anotherchance of a false alarm, so the design that was meant to raise confidence lowers it.
But A/A tests do catch real bugs
The counter-evidence is strong enough that the tree keeps a live A/A on the table. Kohavi, Tangand Xu devote a full chapter to it and their argument is empirical rather than statistical: the ideais useful because the tests fail so often in practice, and each failure forces a team to re-examinean assumption.
Ian Whitestone's write-up lists real catches, including a stats bug that doubledthe false positive rate by skipping a multiple-comparisons adjustment, and redirect side effectswhere bots did not follow the redirect and created a sample ratio problem.
The resolution is not to argue about whether A/A tests work. It is to separate the question the A/Ais being asked to answer from the instrument that answers it cheapest.
Read an A/A on counts, not on conversions
If you do spend traffic on a live A/A, the sample ratio check on user counts is a far more sensitivebug detector than the conversion-rate comparison sitting next to it. Same visitors, sameduration, very different resolving power.
Speero calculation. SRM detection uses a chi-square goodness-of-fit at alpha 0.001 with 80% power. The metric column is the minimum detectable relative difference on a 5% baseline conversion rate at alpha 0.05 with 80% power.
| Users per arm | Allocation skew the SRM check catches | In plain terms | Conversion difference the same test catches |
|---|
| 5,000 | 52.1 / 47.9 | 4.1% of users missing from one arm | 25.9% |
| 10,000 | 51.5 / 48.5 | 2.9% missing | 18.0% |
| 30,000 | 50.8 / 49.2 | 1.7% missing | 10.2% |
| 100,000 | 50.5 / 49.5 | 0.9% missing | 5.5% |
At 30,000 per arm the count check flags a variant quietly losing 1.7% of its users. The conversionreading of the same data cannot see anything under about 10%. That gap is the whole argumentfor making SRM the primary read, and it is also why SRM belongs on every live A/B test, where itcosts no extra traffic at all.
What actually justifies spending traffic on a live A/A
| Trigger | What justifies it |
|---|
| Trigger 01First implementation | New tool, or an existing tool on a site it has never run on. Nothing about the pipeline has been observed end to end yet. |
| Trigger 02Migration | Client-side to server-side, a tool swap, or a new personalisation or CDP layer sitting between the tool and the page. |
| Trigger 03Redirect or split-URL testing | A documented SRM source. Bots and some users do not follow redirects, so one arm loses users before it is ever measured. |
| Trigger 04Numbers that never reconcile | The tool's counts and the analytics counts have never agreed and nobody knows why. That is a plumbing question, and an A/A isolates it. |
| Trigger 05Infrastructure change | New CDN, new consent management platform, a rebuilt tag container, an SPA re-architecture. Any of these can silently break assignment. |
| Trigger 06SRM across multiple tests | A pattern, rather than a single incident, points at the platform. Fabijan et al. call this out explicitly as a diagnostic rule of thumb. |
How common are these failures:
- Roughly 6% of experiments at Microsoft showed a sample ratio mismatch during the KDDstudy period. The authors note that a product running ten thousand experiments a year canexpect to see at least one SRM per day.
- The same paper catalogues 25 distinct root causes across five categories: assignment,execution, log processing, analysis, and interference. This is the taxonomy the diagnosticbranch of the tree points to.
- On the metric side, Microsoft found that the typical product had 10 to 15% of its metrics failing a p-value uniformity check under the null, and for some products the failure rate reached 30%. That is the case for the offline simulated A/A branch.
The three instruments, in cost order
| Cost | Instrument | What it does |
|---|
| Cheapest | Instrumentation check | Zero traffic, hours of work. Does the variant render in both arms, do the events fire, do exposures reconcile with analytics, does bucketing stay sticky across sessions and devices, is there flicker. |
| Cheap | Offline simulated A/A | Zero live traffic, uses history you already have. Resample 1,000+ times and confirm the p-value distribution is uniform. Catches skewed metrics, outlier sensitivity, rare events, and stats-engine bugs. |
| Expensive | Live A/A | Real traffic and real calendar time, and it can only be justified when the whole pipeline is unproven. Read it on counts and plumbing, and treat the conversion-rate p-value as a byproduct rather than the verdict. |
.svg)
