Experimentation / CRO

Experimentation and Testing Programs acknowledge that the future is uncertain. These programs focus on getting better data to product and marketing teams to make better decisions.

Research & Strategy

We believe that research is an integral part of experimentation. Our research projects aim to identify optimization opportunities by uncovering what really matters to your website users and customers.

Data and Analytics

90% of the analytics setups we’ve seen are critically flawed. Our data analytics audit services give you the confidence to make better decisions with data you can trust.

Guide

Using synthetic users in research

Synthetic output is a hypothesis, never evidence. Use it to shape what you ask real people and what you spend traffic on, never in place of either.

Where it fits

Use

Explore

  • Pilot an interview guide or survey
  • Orient in an unfamiliar audience or category
  • Generate hypotheses to probe with real customers
  • Spot unclear claims before a real copy test

Use with care

Focus

  • Drop the weakest variants or concepts before spending traffic
  • Check visual hierarchy before a 5 second test
  • Boost a thin segment using your own real survey data (specialists only)

Don't

Validate, and any stage for these

  • Pick a winner or measure lift
  • Size effects, forecasts or samples
  • Measure intent, usability or NPS levels
  • Supply customer language for copy or keywords
  • Speak for niche, minority or non-Western audiences
  • Judge something genuinely new

Why: synthetic users give you the most common answer

Studies keep finding the same pattern. Synthetic answers land near the typical response and lose the spread of real people, including the unusual words, edge cases and subgroups that research is usually looking for. Researchers call this mode collapse. Rankings tend to hold up better than sizes, which is why synthetic methods can help you prioritise but can't tell you how much something will move.

Five questions before you run one

  1. What decision will this change, and what does a wrong answer cost?
  2. Is this about real behaviour or what people say they think?
  3. Is the audience well represented in public data?
  4. Do we need a ranking, a number or the full range of answers?
  5. Which real-people method follows it, and when?

Rules every time

Pair it with real people

Every synthetic step is followed by a method that could prove it wrong.

Keep it upstream

Synthetic methods carry a low strength of signal and never sit in a Validate step.

Read the order, not the size

Trust which option ranks higher more than how big the gap looks.

Expect flattery

Discount agreement, and treat a narrow spread of answers as a trait of the model.

Ground it in your data

Personas built from your own interviews beat demographic prompts.

Label it

Reports say which findings came from people and which came from models.

Synthetic and hybrid methods in the hub

Sources: AAPOR Task Force on AI in survey research (2026), Nielsen Norman Group (2024 and 2026), Insights Association, MRS, and published studies on LLM survey and A/B prediction. Last reviewed 16 November 2026.