Experimentation / CRO

Experimentation and Testing Programs acknowledge that the future is uncertain. These programs focus on getting better data to product and marketing teams to make better decisions.

UX Research

We believe that research is an integral part of experimentation. Our research projects aim to identify optimization opportunities by uncovering what really matters to your website users and customers.

Data and Analytics

90% of the analytics setups we’ve seen are critically flawed. Our data analytics audit services give you the confidence to make better decisions with data you can trust.

3 Methods for Confirming Test Effects

3 Methods for Confirming Test Effects

Is the result you found in a test repeatable. THIS is the gold standard of scientific research.

These are common methods of cross-validation. Note that holdouts can be difficult to maintain, results also have to be accurate and there can be reliability issues there. It also sacrafices from the solution benefit while running. Continuous holdouts can on the other hand lose attribution of any false positives, but they are easier to maintain.

Alternatives would be to implement fully and reverse-test the control on a portion of traffic, or to implement and do a time-bucket comparison, a longitudinal test.

Use Cases

- need to report out on the ROI of a test initiative or group of tests
- need to be extra confident in the test result

Type:
Ritual
Pillar:
Decision & Execution

Get this Blueprint PDF
Get the Figma Toolkit
Toolkit includes PDF of the blueprint, links to related assets and a loom video on using the blueprint.

No spam - just the good stuff

Get this blueprint toolkit and the best of experimentation and CRO news delivered right into your mailbox, every week!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Open Figma Toolkit Now
Toolkit includes PDF of the blueprint, and if available links to related assets and a loom video on using the blueprint.