Why the default answer is no
An A/A test is usually asked for as a trust exercise. The problem is that the version people run, two identical experiences read on a conversion-rate p-value, is the weakest instrument available for the job. Three things go wrong with it.
A pass proves almost nothing. A non-significant A/A does not demonstrate the tool is sound, but only demonstrates the test lacked the power to find a difference. To rule out a difference as small as 0.1% relative on a 5% baseline conversion rate you would need roughly 298 million visitors per arm at 95% confidence and 80% power. AB Tasty put the same figure at around 300 million.
A failure is expected. At a 0.05 threshold, one A/A in twenty comes back significant while everything is working correctly. Teams that read that as a broken tool then distrust results they should have believed. Georgi Georgiev's objection is exactly this - the significance threshold already accounts for the outcome, so a rare significant result is a predictable event, not evidence of a fault.
Adding arms makes it worse. A/A/B and A/A/B/B designs are often proposed as a way to get validation and a real test in one shot. Every extra arm is another comparison and another chance of a false alarm, so the design that was meant to raise confidence lowers it.
But A/A tests do catch real bugs
The counter-evidence is strong enough that the tree keeps a live A/A on the table. Kohavi, Tangand Xu devote a full chapter to it with an empirical (not statistical) argument - the idea is useful because the tests fail so often in practice, and each failure forces a team to re-examine an assumption.
Ian Whitestone's write-up lists other catches, including a stats bug that doubled the false positive rate by skipping a multiple-comparisons adjustment, and redirect side effects where bots did not follow the redirect and created a sample ratio problem.
The resolution is not to argue about whether A/A tests work. It is to separate the question the A/Ais being asked to answer from the instrument that answers it cheapest.
Read an A/A on counts, not on conversions
If you do spend traffic on a live A/A, the sample ratio check on user counts is a far more sensitive bug detector than the conversion-rate comparison sitting next to it. Same visitors, same duration, very different resolving power.
Speero calculation. SRM detection uses a chi-square goodness-of-fit at alpha 0.001 with 80% power. The metric column is the minimum detectable relative difference on a 5% baseline conversion rate at alpha 0.05 with 80% power.
| Users per arm | Allocation skew the SRM check catches | In plain terms | Conversion difference the same test catches |
|---|
| 5,000 | 52.1 / 47.9 | 4.1% of users missing from one arm | 25.9% |
| 10,000 | 51.5 / 48.5 | 2.9% missing | 18.0% |
| 30,000 | 50.8 / 49.2 | 1.7% missing | 10.2% |
| 100,000 | 50.5 / 49.5 | 0.9% missing | 5.5% |
At 30,000 per arm the count check flags a variant quietly losing 1.7% of its users. The conversion reading of the same data cannot see anything under about 10%. That gap is the whole reason for making SRM the primary read, and it is also why SRM belongs on every live A/B test, where it costs no extra traffic at all.
What actually justifies spending traffic on a live A/A
| Trigger | What justifies it |
|---|
| Trigger 01First implementation | New tool, or an existing tool on a site it has never run on. Nothing about the pipeline has been observed end to end yet. |
| Trigger 02Migration | Client-side to server-side, a tool swap, or a new personalisation or CDP layer sitting between the tool and the page. |
| Trigger 03Redirect or split-URL testing | A documented SRM source. Bots and some users do not follow redirects, so one arm loses users before it is ever measured. |
| Trigger 04Numbers that never reconcile | The tool's counts and the analytics counts have never agreed and nobody knows why. That is a plumbing question, and an A/A isolates it. |
| Trigger 05Infrastructure change | New CDN, new consent management platform, a rebuilt tag container, an SPA re-architecture. Any of these can silently break assignment. |
| Trigger 06SRM across multiple tests | A pattern, rather than a single incident, points at the platform. Fabijan et al. call this out explicitly as a diagnostic rule of thumb. |
How common are these failures:
- Roughly 6% of experiments at Microsoft showed a sample ratio mismatch during the KDD study period. "A product running ten thousand experiments a year can expect to see at least one SRM per day."
- The same paper catalogues 25 distinct root causes across five categories: assignment, execution, log processing, analysis, and interference. This is the taxonomy the diagnostic branch of the tree points to.
- On the metric side, Microsoft found that the typical product had 10 to 15% of its metrics failing a p-value uniformity check under the null, and for some products the failure rate reached 30%. That is the case for the offline simulated A/A branch.
The three instruments, in cost order
| Cost | Instrument | What it does |
|---|
| Cheapest | Instrumentation check | Zero traffic, hours of work. Does the variant render in both arms, do the events fire, do exposures reconcile with analytics, does bucketing stay sticky across sessions and devices, is there flicker. |
| Cheap | Offline simulated A/A | Zero live traffic, uses history you already have. Resample 1,000+ times and confirm the p-value distribution is uniform. Catches skewed metrics, outlier sensitivity, rare events, and stats-engine bugs. |
| Expensive | Live A/A | Real traffic and real calendar time, and it can only be justified when the whole pipeline is unproven. Read it on counts and plumbing, and treat the conversion-rate p-value as a byproduct rather than the verdict. |
.avif)
.svg)
