Author: Martin P.
Title: Content Marketer
EP 36: 3 Methods for Confirming Test Effects
We love experimentation. That's where the gold nuggets come from — Reggie Fils-Aime
Yo yo,
Martin P. from Speero here.
This time, no long intros.
Let’s jump straight in
This Week in Experimentation:
Blueprint of the week: 3 methods for confirming test results. Is the result you found in a test repeatable? THIS is the gold standard of scientific research. Link.
Talk of the week: How can you create an A/B testing framework? Kameleoon did an interview with our Lucia van den Brink right on this topic. Link.
Read of the week: How to be a good Experimentation LEADER with Cristina Pujol, by Experiment Nation. Learn what’s the experimentation ambassador role, the experimentation process for domains, and more. Link.
Event of the week: Emerce Conversion and Analytics — 350 attendees max, 4 plenary keynotes, 28 parallel sessions in 4 tracks. Link.
Blueprint of the Week: 3 methods for confirming test results
Is the result you found in a test repeatable? THIS is the gold standard of scientific research.
These are 3 common methods of cross-validation.
Note that holdouts can be difficult to maintain, results also have to be accurate and there can be reliability issues there. It also sacrifices from the solution benefit while running.
Continuous holdouts can on the other hand lose attribution of any false positives, but they are easier to maintain.
Alternatives would be to implement fully and reverse-test the control on a portion of traffic or to implement and do a time-bucket comparison, a longitudinal test.
Use Cases:
— Need to report on the ROI of a test initiative or group of tests
— Need to be extra confident in the test result
Talk of the Week: How to be a good Experimentation LEADER with Cristina Pujol
Experiment Nation brought Cristina Pujol from Vista to explain why being an experimenter and experimentation require radically different skills. Here’s what you can expect:
— How Vistaprint organizes for Experimentation
— The Ambassador role (the link between the experimenters in its domain and the experimentation hub)
— Experimentation process for domains (backlog of experiment ideas, idea review with analytics, creating testing plan in public, and more)
— Ideas to share learnings and celebrate success
Read of the Week: How can you create an A/B testing frame
How can you create an A/B testing framework? Kameleoon did an interview with our Lucia van den Brink right on this topic.
They also discussed:
What frameworks do you need for an effective experimentation program?
— Pre-test
— Test Ideation
— Prioritization
— Statistics
Why do you need A/B testing frameworks if you already got a process?
Top five experimentation frameworks:
— Strategic Testing Framework
— RXL
— Learning, speed, velocity
— Evidence-based prioritization framework
— User research methods framework
Link.
.svg)
