Author: Martin P.
Title: Content Marketer
EP 48: Experimentation Phasegate Blueprint
Experimentation is a big part of how you find the best of the best — Kari Skogland
Hey hey.
Martin P. from Speero here.
Only one thing on my mind.
Does this newsletter format work?
Or are the emails still long?
Reply and tell me.
Without further ado, it’s time to see…
This Week in Experimentation:
Blueprint of the week: Experimentation Phasegate — determine the cadence and flow of your experimentation flywheel. Link.
Talk of the week: Testing Insights with Brianna Warthan from AT&T — Sometimes, experimentation is about coaching, allowing freedom to fail, and providing helpful, usable data back to the org. Link.
Read of the week: Form Design by Coop — a clean and slick design system for making forms, for us non-designers. Link.
Event of the week: MeasureCamp Paris 2023 — This year’s themes include data collection, analysis, governance, visualization, measurement, and regulatory collection constraints. Link.
Speero Story of the week: Anchor the Bigger Price — how Speero used anchoring bias to increase revenue per visitor and AOV (read below).
Blueprint of the Week: Experimentation Phasegate Framework
Experimentation Phasegate is probably our biggest, boldest, and craziest blueprint we got. It should help you answer the question:
— What are the Phases of an experiment?
— What are the stages?
— What is the process?
— What does the process look like?
Experimentation Phasegate also helps you ask vital questions from yourself and your team during different stages of an experiment:
— From collecting data and getting an insight
— From turning that insight into a business case
— Into a study design
— Into the actual planning and building of the experiment
— Into testing and launching and reporting.
This is the belly of the beast.
It describes the activities, questions, deliverables, use cases, and gates of the ritual of processing through the experiment.
It also references the artifact in terms of building a test document and having all of those pieces sorts of stack up with each other.
It highlights the key ingredients of the test plan and scorecard being like:
— What are you gonna do?
— Why are you doing it?
— Who’s doing what?
— What is the result and conclusion of what you've done and the data that you've collected?
The best thing about this blueprint is the gate itself. When you set up programs or when you set up the flywheels of experimentation, these become one-way gates with no turning back. Speero has a Miro board for this that is linked to the blueprint page.
Use Cases:
— determining the cadence and flow of the flywheel
— understanding what/who/where of the flywheel
— using it as a communication tool to align the team on how things work
Talk of the Week: Testing Insights with Brianna Warthan from AT&T
Sometimes, a great testing program functions more like a coaching program. At least, that’s what it looks like at AT&T.
On this episode of Testing Insights, experimentation Leader Brianna Warthan explained that her team provides lots of advice to their product managers and strategic partners.
They also allow freedom to fail.
If a test is a loss, they guide them through the process for how to do it better next time.
Ultimately, their goal is to provide helpful, usable data back to the organization.
That means every time they encounter a new hypothesis, they tailor the methods to stakeholder needs.
To hear exactly how Brianna’s team approaches testing and sets guardrails, tune in to this episode of Testing Insights.
Read of the Week: Form Design by Coop
How should you design your forms? Experimenters often don’t come from design backgrounds, so this is a unique challenge for most of us.
We don’t want to spend time learning secondary stuff but on another hand… you need stuff that looks great and doesn’t anger your users.
That’s why the read of this week is Form Desing by Coop. Check out the page. They’ve made a clean and slick design system for making forms, including:
— Form elements
— Layout and hierarchy
— Grouping, structure, and flow
— Labels and input fields
— Buttons
Event of the Week: MeasureCamp Paris 2023
MeasureCamp is an “ unconference ” held all Saturday, bringing together hundreds of professionals: advertisers, consultants, publishers and agencies. Unlike traditional conferences, the talks are proposed the day of the event, and delivered directly by the participants. Everyone is encouraged to participate by proposing and leading sessions/workshops on the theme of their choice. The possible topics for this event include:
— Measurement, collection & analysis of data
— Data governance
— Regulatory collection constraints (eg: GDPR), browsers (eg: ITP 2.x)
— Allocation/contribution
— Data visualization & BI
— Personalization & Activation
— Path and Conversion Optimization (CRO)
Saturday Jun 24, 2023 - 9:00 a.m. - 5:00 p.m. + after
HETIC, 27 Bis Rue du Progrès, Montreuil 93100, Paris
Speero Story of the Week: Anchor the Bigger Price
What if you show the higher price on the product page? This is exactly what Speero did with one of our clients. A simple test that turned out great results. Here’s how it went.
We all occasionally fall into anchoring bias — relying heavily on the first piece of info we see when we make decisions.
For instance, a visitor who first sees a high-price product might think that everything else is relatively inexpensive, even if other products are actually expensive.
This is the first test we ran for this client. So, we thought about the previous learnings we got with prior clients like Codecademy.
This was their product page.
It defaults to deluxe.
We wanted to anchor to the higher price.
A super simple test to set up.
But the test did quite well.
🔷 — Test Plan Stage:
Hypothesis:
If we order the size options from highest to lowest it will anchor users to the grand option, which will raise the value perception of the default deluxe choice and lead to more users choosing a larger bouquet and higher overall AOV.
🔷 (3) — Scorecard Stage:
Winner! Variation resulted in +5.2% Revenue per Visitor & +1.5% AOV at 99% significance. We recommend implementing the Variation.
Next steps/Iterations:
1) Highlight the “best value” product.
2) Add a small review of someone loving the “Deluxe” package.
3) Explain how much more value they get with the most expensive option.
.svg)
