Author: Martin P.
Title: Content Marketer
EP 20: Sample Ratio Mismatch Error
Test ideas. Then follow them.
Welcome to Speero’s newsletter: The Experimental Revolution.
Where we blast blueprints, opinions, news, and trends that will revolutionize the way you experiment.
As always, Martin P. is behind this screen.
Here’s this week at a glance.
This Week in Experimentation:
Blueprint of the week: Sample Ratio Mismatch Error. A blueprint for helping solve SRM errors both from the client and server sides. Link.
Talk of the week: Problem-based hypothesis for AB/MTT testing experimentation: Oliver Paton takes a guest from Speero to discuss the missing link between making a hypothesis and research. Link to video.
Read #1: Most product ideas don’t work or make your product even worse. Explore 4 groups of validation methods for testing your ideas, instead of randomly publishing them. Link.
Read #2: Pancakes and experimentation — Learning by doing. Your first experiment won’t be great. But it's a necessary step toward building your first experimentation program. Link.
Opinion of the week: CRO is not the same as CXO. Both aren’t experimentation. Link.
Event of the week: Learn how to apply user research to your experimentation program and take an always-on attitude about your users’ needs and pain points. Link to register.
Blueprint of the Week: Sample Ratio Mismatch Error
You got a Sample Ratio Mismatch alert! What do you do?
You’ve set the ratio for your test at 50/50 and your testing tool has reported that you have a 70/30 traffic split in your experiment.
You can’t trust this data.
You gotta rectify it before you analyze the results.
There are several ways that your experiment can have a sample size mismatch, but the most common will be technical issues with the segmentation itself.
This blueprint is your guide for finding the best next step once you get an SRM error.
Use Cases:
You have received a Sample Ratio Mismatch (SRM) error from your testing tool or analytics team.
Talk of the Week: Problem-based hypothesis for AB/MTT testing experimentation
Oliver Paton, AVP Digital Analytics & Optimisation capability lead, has recently started a YouTube channel about CRO.
In his latest episode, he and Shiva Manjunath talked about the missing link between making a hypothesis and research.
If you add this as a pivot point in your process you can gain many advantages, which they go through in the video.
Reads of the Week:
Read #1: Idea Validation — Much More Than Just A/B Experiments. Link.
Here’s what you should know about product ideas:
- The vast majority of them have no measurable impact on value-to-customer or business results. Some are even negative.
- No one, however experienced or talented, can predict the ideas that will work. Too many unknowns.
- Most businesses employ weak heuristics, archaic decision processes, and opinions to make bets on a dozen of unproven ideas.
This is likely the biggest source of waste in the industry.
However, there are four ways to reduce this.
So-called four types, or levels, of idea validation methods:
- Assessment
- Fact Finding
- Tests
- Experiments
Explore how to use them in your product design. Link.
Read #2: Pancakes and experimentation: Learning by doing. Link.
Starting an experimentation program is like making pancakes. Your first foray will take a lot of time and produce doubtful results. But however the first experiment turns out ugly or imperfectly cooked, you have to start with a trash pancake for pancakes number two through n to be possible. Even good. Link.
Opinion of the Week:
By Martin P.
“CRO is not the same as CXO. Both are not the same as Experimentation.
CRO was a misnomer to start with.
It pushed the whole industry toward revenue and conversion wins.
But this isn’t the whole picture.
It never was.
You need to think about how you make your customers feel.
All those pop-ups may increase the profit, but…
They can also push visitor away after a while.
Where do you draw the line between revenue and customer satisfaction?
This is where CXO comes into play.
You take customer experience into consideration.
Add it to quick wins in the program’s goals.
But we still have a way to go.
Experimentation ≠ CXO
Experimentation isn’t only what you learn about customers from testing.
It’s how you test as well.
How fast do you test?
How many tests are you churning out?
How do you decide what to test first?
In the end, experimentation is about testing any kind of business decision.
Not just features, new releases, or website or app changes.
You can test anything.”
Link.
Event of the Week: Experimentation Elite
If you want to address real user needs and pain points, you need always-on user research. If you wanna provide consistent value, you need user research.
If you want to lower the risk of building the wrong stuff or building the right stuff in the wrong way, you need to… You get the drift.
Luckily, Emma Travis, director of research at Speero, is going to talk on Experimentation Elite precisely about that:
— How to add always-on user research to your experimentation program —
Expect to learn and get:
- How to turn customer insights into an actionable roadmap
- Examples of "always on" research in practice
- Frameworks & templates to help you to get started
- How to use research methods alongside experimentation to support informed business decision.
Leverage user research to address the right questions and the right needs with the right features at the right time.
Happening on 8th December 2022, at our good friend's Experimentation Elite.
.svg)
