Research method
Preference Test
Preference testing involves showing participants two (or more) versions of a page, an element, or a piece of copy, and asking them to select their preference and provide rationale for their preference. Data from the full sample of participants can then be used to draw conclusions about the best-performing version.
When to reach for it
Preference tests are a validation technique designed to gather data on a concept or solution. As such, preference tests can be beneficial in the planning stages of an AB test to either pre-validate or refine test concepts. Preference tests can also be used as an alternative to an AB test where true AB testing isn't possible for technical reasons or where there's not enough traffic.
Strengths and limits
Pros
Preference testing is generally quick and inexpensive, providing a quick and easy way to gather data to help inform concept design.
Cons
Participants can't interact with functionality in a preference test. This means feedback is focused on the design, look and feel of the page, rather than being a true assessment of ease of use. This makes the method more suitable when the required feedback is based on the design or page copy, rather than the functionality or ease of use.
Despite similarities with a true AB test, preference testing is subject to some research bias, since participants are not truly visiting your website or parting with their data/money.
The low time investment from participants and thus the low level of incentive can impact the quality of the responses. It's important to review any qualitative feedback to help identify void responses.
Practical considerations
Sample Size
Preference tests gather quantitative data, as such the sample size should be large enough to draw robust conclusions. We would recommend including at least 30 participants (ideally more).
Tools
There are many preference testing tools. Typically this is an option built into tools like Maze, Usabilityhub and Wynter, rather than there being stand-alone tools. Maze and Usabilityhub are focused on design preference tests, whereas Wynter is focused on copy preference tests.
Common tools
Go deeper
Recipes that use this method
A method earns its value when its signal corroborates others. Here is where this one does its work.
Last updated
.svg)
