Research method
A/B Testing
- Validate
Running a live, randomised experiment that splits traffic between a control and one or more variants to measure the causal effect of a change on a target metric.
When to reach for it
Confirming that a change actually improves the outcome it targets before rolling it out fully. Settling disagreements about a design or flow with real customer behaviour rather than opinion.
Pros
Gives a direct, causal read on impact rather than a proxy signal. Removes bias from the decision since real customers self-select into each path.
Cons
Needs enough traffic and time to reach a reliable result. Only tests the specific variants built, so it depends on earlier research to shape a strong hypothesis.
Practical considerations: sample sizes, tools, logistics
Practical considerations
Best run once a hypothesis has already been shaped by exploratory and focus-stage research, since a test is only as good as the idea behind it.
Never in isolation
Recipes that use A/B Testing
A method earns its value when its signal corroborates others. Here is where this one does its work.
Compare the platforms that run these tests, with pricing, SDKs and warehouse support side by side. A/B Testing Tools Comparison
Last updated
.svg)
