Experimentation / CRO

Experimentation and Testing Programs acknowledge that the future is uncertain. These programs focus on getting better data to product and marketing teams to make better decisions.

Research & Strategy

We believe that research is an integral part of experimentation. Our research projects aim to identify optimization opportunities by uncovering what really matters to your website users and customers.

Data and Analytics

90% of the analytics setups we’ve seen are critically flawed. Our data analytics audit services give you the confidence to make better decisions with data you can trust.

3 Methods for Confirming Test Effects Blueprint

Is the result you found in a test repeatable?

THIS is the gold standard of scientific research. 3 Methods for Confirming Test Effects Blueprint provides the most common methods of cross-validation. 

Note that holdouts can be difficult to maintain, results also have to be accurate and there can be reliability issues there. It also sacrifices the solution benefit while running. On the other hand, continuous holdouts can lose attribution of any false positives, but they are easier to maintain.

The first alternative is flip tests, where you implement the winner and then rerun the test by removing the winner. Flip tests are probably the easiest to implement and most common to use, especially on a test-by-test basis. But, they got a burning question inside. What if it loses? 

Going backward can sometimes erode trust in your program. For example, when it gets flat, loses, or gets a different type of result. But this is a part of flip tests. If you’ve got a good program and experimentation culture to handle that, you’ll be fine.

The last solution is time series and moving averages. The point of time series and moving averages is that you implement a test and see what happens over time. But you gotta be careful. There are lots of confounding variables here. You can try using the GA effect tool that allows you to do this more academically

Use Cases:

- Report on the ROI of a test initiative or group of tests.
- Be extra confident in your test result.

Decision & Execution

Get this Blueprint PDF
Get the Figma Toolkit
Toolkit includes PDF of the blueprint, links to related assets and a loom video on using the blueprint.

No spam - just the good stuff

Get this blueprint toolkit and the best of experimentation and CRO news delivered right into your mailbox, every week!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Open Figma Toolkit Now
Toolkit includes PDF of the blueprint, and if available links to related assets and a loom video on using the blueprint.