Welcome to the first edition of Briefly Experimental
Every two weeks we'll deliver the best experimentation content and commentary, curated by a member of the Speero team.
In each edition, we'll break things down into the four key pillars needed for any successful experimentation program;
- Strategy & Culture
- People & Skills
- Process & Methodology
- Data & Tools
This edition was written by Ben Labay, MD of Speero.
Strategy & Culture
👀 A peek inside the experimentation program at Farfetch
I was lucky enough to have Luis Trindade, Principal Product Manager, Experimentation at Farfetch as a guest on the Testing Insights show. Luis and I discussed how Farfetch built its own testing tool and the rituals they use to get all areas of the business involved in experimentation, from those working on backend code to their B2B2C platforms.
Watch the Testing Insights episode with Luis Trindade here.
Luis and his colleagues are also super generous when it comes to sharing what they’re doing. Check out their guide to running experimentation - which has influenced my own education on experimentation program functions.
🆕 Marketers prioritize new customers over old and it's costing them
According to a new Nielsen report (signup required), customer retention was the top marketing priority for only 13% of respondents while preventing churn was only deemed of top importance by just 8%. What is with Marketing's obsession with acquisition? What happened to the 80/20 rule?
HBR estimates the cost of acquiring a new customer is 5-25 times more expensive than retaining your existing ones. While a 5% increase in customer retention can lead to 25% - 95% more profit. Now is a good time to refocus marketing objectives and budgets to address the above.
People & Skills
🧏🏻♀️ How Microsoft’s Chief Accessibility Officer does her job
CAO is a rare job title and it caught my attention because accessibility is so often an after-thought (if at all) when it comes to creating any type of experience. This is a nice interview with Jenny Lay-Flurrie about her role at Microsoft and the policies they have in place to hire more people with disabilities.
This comment also stuck out;
"I don’t think anyone could have quantified the impact of talking books when they were created for the blind. And closed captioning [automatic subtitles] went up for us on Teams significantly between February and April last year. That wasn’t just the deaf community."
An interesting point; in making technology, products, or experiences accessible, it doesn't just improve things for those with specific needs, but for everyone - overall the experience will be better.
Process & Methodology
🧪 Spotify’s new experimentation coordination plans
How do maintain the integrity of a test and the user experience if you’re running 100s of experiments for hundreds of millions of users at the same time?
The Experimentation Platform team at Spotify has done some great research into different approaches for testing at scale and set out their chosen approach; Bucket Resue.
👩💻 Adobe created the CARE framework to help design enterprise software
A nice little framework from Adobe to help engineering teams keep customer needs in mind when building new products or features.
CARE stands for;
C - Customer needs drive all your work, so focus on the “why” instead of “what”
A - Adoption paths require planning and consist of many steps
R - Recognize that the happiest scenario is not all there is
E - Enable teams that support your customer and product
Data & Tools
☠️ "A test program under the data team leads to mental masturbation"
A recent chat with Simon Elsworth, Senior Experimentation Manager at Sky, inspired the quote above.
Well, there’s a very cool study that set up 73 research teams with the same data, to test the same hypothesis.
All over the place.
Data can be dangerous, and information with a complex process of interpretation can lead you to very different outcomes, as the researchers found;
"This study provides evidence of a vast universe of research design variability normally hidden from view in the presentation, consumption, and perhaps even creation of scientific results."
The answer to this is to focus on big problems. The data should be intuitive if you're focusing on clear problems. Focus on the questions, better questions will lead to better data.
This is the first edition of Briefly Experimental
We have a lofty goal. We want Briefly Experimental to become something you genuinely look forward to arriving in your mailbox. A source of the most interesting and thought-provoking articles related to experimentation program management.
But in order to do that we need your feedback. While a ton of work went into curating this we know we can make it even better. So send your thoughts and feelings to firstname.lastname@example.org.