Author: Martin P.
Title: Content Marketer
EP 12: Running Multiple Experiments Simultaneously?
Keep experimenting. Keep fighting the good fight.
Yo. Martin P. in the (email) house.
To all new subscribers, welcome.
If you didn’t catch up, you can read previous emails here.
As for oldtimers, got anything you want me to cover?
Just reply to this email.
Blueprint of the Week: Running Multiple Experiments Simultaneously
Worried about how running an A/B test on your product page impacts another experiment in the checkout?
How much data pollution do you get on your hands when running multiple A/B tests at the same time?
How harmful is the interference?
Planning test runtime might have you think about these questions. But as Lukas Vermeer (Director of Experimentation at Vista, ex-Booking.com) puts it:
‘Overlapping experiments are the least of several evils.’
How come?
Well, for starters, interaction effects are rare.
And if there’s an interaction effect between your tests, you can still detect them.
Even more, interaction effects can inform you since they provide new information to the table—cross effects of multiple tests.
Read the whole article here.
Get your free template here.
Talk of the Week: The True Value of Experimentation
As the Product Lead at Loblaw Digital, Rommil Santiago helps oversee hundreds of tests per year.
His role encompasses experimentation, personalization, SEO, analytics, and more.
The first-ever guest on our new “From A to B” series, Rommil took the time to share in great detail the value of experimentation and why experiments go beyond the quick wins.
In the video, Rommil shares:
— Why experimentation is more than a check-the-box endeavor
— The relationship between experimentation and research
— Why you should start by chasing low-hanging fruit
— When to layer in a “test to learn” approach
— How to shift the focus from revenue to learnings
— How to communicate with executives
Reads of the Week:
The theory, the practice, and the pitfalls of A/B testing: Take a deep-dive into the theory of A/B testing of Machine Learning Models. You’ll learn:
— How to split your traffic for A/B testing,
— What it means for the test result to be statistically significant,
— The possible test results and the 4 types of test errors, and
— Common pitfalls to avoid.
Link.
6 steps to consider when planning a systematic set of A/B tests to optimize a design for the long haul: Have you ever discussed design options with your team, and everyone was split on the best way forward?
After a few minutes, someone always suggests doing an A/B test to select the best option.
The issue is, testing takes a while. In low traffic situations, you may need to wait weeks and months for results. Even worse, what to do when test results are inconclusive? Nielsen Norman group presents a better way. Link.
Pricing is your most powerful lever for fast growth. But it’s also one of the most mysterious levers out there, with not a lot of info on the topic.
Kyle Poyar from OpenView and pricing expert and quant wunderkind Abel Riboulot, Co-Founder and CEO of Corrily, have made one of the best guides on pricing tests.
Explore why you might be widly underpriced without even knowing it. Link.
.svg)
