‍Author: Martin P.
Title: Content Marketer

EP 49: Should I Run Experiments Simultaneously?

Yo.

Martin P. from Speero here.

I asked — you sent feedback.

We’re changing the format of the newsletter.

From now on, I’ll write a short summary of the best reads, talks, and events just below.

Blueprints and stories (as special sections) will remain but will be shorter as well.

That’s it. Reply if you like the new format (so I know I’m doing this right 😀)

This Week in Experimentation:

Blueprint of the week: Should I run Experiments Simultaneously? (Generally) Yes, you should. If you don't you face multiple worse 'evils'.

Talk of the week: A/B Testing Statistics Concepts Experimenters must know with Ronny Kohavi

Learn A/B Testing Statistics, how to use the Overall Evaluation Criterion, Twyman's law, how much traffic you need to run A/B tests, and other statistical concepts all Experimenters must know.

Read of the week: A Review of Propensity Score Modelling Approaches — learn what is propensity score and how they’re used, including the 3 most common methodologies.

Propensity score modeling allows you to stratify your treatment and control to remove behavioral and demographic biases that might be acting as confounders

A propensity score is the probability that an individual will be exposed to the treatment. If we took 100 identical customers and 70 of them were exposed to the treatment then they would all have a propensity score of 0.7

Event of the week: CXL Live 2023 (in Speero + CXL organization) — the top experimentation and growth event that’s about relationships, not powerpoints. 

This event focuses on building authentic relationships and spending quality time with other like-minded people. You are guaranteed to meet people with a similar role at a similar company working on the same challenges that you are.

That’s not to say content doesn’t matter: it does, and we’ve always brought our A-game. But high-quality content is table stakes. And in the end, other things matter more. 

Speero Story of the week: Tree Testing a New Navigation — Speero’s client is a B2B company in the Healthcare sector. Their navigation was confusing their website visitors. AB testing was not possible due to technical limitations. Speero Tree Tested new navigation.

Blueprint of the Week: Should I run Experiments Simultaneously?

(Generally) Yes, you should. If you don't you face multiple worse 'evils':

1) Sequential Isolation issue, drastically reducing testing velocity, and thus learning rate, test impact, and the number of tests you run.

2) Traffic or Parallel Isolation Issue — isolate your traffic, and you drastically reduce your statistical power and increase the testing time.

3) Now you don’t know their interaction effect, which means you need a third, combining test to measure it.

The last ‘evil’ is making changes to your website or app without testing. In other words, going blind.

Blueprint Use Cases:

1) Define the strategic KPIs that fuel optimization efforts

2) Identify metrics that are crucial to the company's success

3) Break down KPIs into smaller metrics where we can see improvements faster

Speero Story of the Week: Tree Testing a New Navigation

The client is a B2B company in the Healthcare sector.

Their navigation was confusing their website visitors.

Users had trouble understanding navbar categories.

AB testing was not possible due to technical limitations.

We developed a new main navigation and decided to validate the proposed changes with tree testing.

Plan:

— Run via Optimal Workshop

— 50 participants per test 

— 6 tasks based around core user journeys

— Two qualitative post-task questions

— Data analyzed; first click, task success (direct & indirect), destination. 

The result?

Avg. task success rate improved by +164%.

Average Task Success: 

Control - 2.2

Variation - 5.8

Recommendation: Implement the variation.