Author: Martin P.
Title: Content Marketer
EP 45: Program Metrics
“The Six Steps to Success: 1) Define Success 2) Devise a Plan 3) Execute and Overcome Adversity 4) Measure Results with Key Metrics 5) Revise the Plan 6) Work Hard”
― Ken Poirot
A new week, a new Speero email.
Hi everyone.
Martin P. from Speero here.
The old format is back.
No maturity, just links leading to good reads, talks, blueprints, and events.
So, what’s the look of…
This Week in Experimentation:
Blueprint of the week: Program Metrics — The KPIs you need to track the success of your program and improve its efficiency and effectiveness. Link.
Talk of the week: How to Leverage Disruption in Experimentation — Or how to have a more strategic, disruptive approach in your experimentation program. Link.
Read of the week: The Negative Test — an original way to test previously baked assumptions and multiple changes. Link.
Event of the week: Accelerating Innovation With A/B Testing — get to know the technical and cultural challenges in creating and running trustworthy controlled experiments. Link.
Speero Story of the week: An eCommerce client of ours sells hundreds of supplements. Now, they came up with an obvious goal for 2023: Improve AOV, profitability, and retention. How did we do this? Read below.
Blueprint of the Week: Program Metrics
You shouldn’t just track individual experiment goals and metrics.
You should monitor the overall success of your program as well.
Program Metrics blueprint helps with just that.
The metrics in it help you monitor the factors impacting the success of your experimentation program, and thus the results you can generate.
Use cases:
— Monitor the success of your experimentation program.
— Increase testing velocity.
— Improve the efficiency and effectiveness of the testing program.
Talk of the Week: How to Leverage Disruption in Experimentation
Speero’s CEO, Ben Labay, was a guest in the latest season of 1000 experiments club, an AB Tasty’s experimentation podcast featuring people who ran over, you guessed it, 1000 experiments.
The main topic was how to have a more strategic, tactical, and disruptive approach to experimentation. Three main takeaways:
1) Portfolio management of experimentation — tag experiments on the solution spectrum (the blueprint from our last email) with the following tags: Iterative, Substantial, Disruptive.
2) Let your Goal Tree Map guide you — create topics based on multiple sources of data you collect. The key here is thematic analysis.
3) Build engagement and efficiency into your experimentation strategy — break down silos between operations and development to enhance collaboration and efficiency between the teams.
Read of the Week: The Negative Test
We typically focus on measuring the impact of adding something to the user experience. But what if you took a completely opposite approach?
Remove something from the user experience, or intentionally make the experience worse?
Welcome to negative testing, where you remove parts of or worsen the UX so you can understand the impact (both positive and negative) of the parts you worsened/removed.
Negative Tests are dramatically useful to test assumptions you already baked into your product. For example, is the search bar really that useful?
You can also use them to test the impact of lots of small changes over time, like improvements to site speed.
Event of the Week: Accelerating Innovation With A/B Testing
It’s time to become humble.
We’re terrible at accessing the values of ideas.
Through several examples and real stories, Ronny Kohavi will show you the humbling reality of the surprising experiments he ran at Microsoft, Airbnb, and Amazon.
You’ll get to know the technical and cultural challenges in creating and running trustworthy controlled experiments, including OEC (Overall Evaluation Criterion), scaling, hurdles, Twyman’s law, and more.
.svg)
