3 Methods for Confirming Test Effects

Is the result you found in a test repeatable. THIS is the gold standard of scientific research.

These are common methods of cross-validation. Note that holdouts can be difficult to maintain, results also have to be accurate and there can be reliability issues there. It also sacrafices from the solution benefit while running. Continuous holdouts can on the other hand lose attribution of any false positives, but they are easier to maintain.

Alternatives would be to implement fully and reverse-test the control on a portion of traffic, or to implement and do a time-bucket comparison, a longitudinal test.

- need to report out on the ROI of a test initiative or group of tests
- need to be extra confident in the test result