Welcome to Briefly Experimental
This edition was written by John Mills, Director of CRO at Speero.
Every two weeks we'll deliver the best experimentation content and commentary, curated by a member of the Speero team. We'll break things down into the four key pillars needed for any successful experimentation program.
Strategy & Culture
⏳ Longer-term strategies to get through the bad times
If you want to weather the downturns, you have to spend more time and energy on building sustainable business growth, through improving retention and CLTV.
While downturns often lead to increased pressure on acquiring customers to cover any revenue shortfalls, or in our case increasing conversion rates of those new customers, it's not always the best plan. Here's what can happen if you aren't measuring things after the initial purchase;
When the cost of acquisition outweighs the lifetime value of a customer, businesses enter a downward spiral – the zone of destruction, each new customer acquired can end up costing the business.
Instead of trying to patch over any shortfalls, focus your efforts on moving the needle for customers you already have. How can you increase their loyalty to your business? How can you increase their CLTV?
One of the interesting considerations raised in the piece by my colleague Annika Thompson and Convert's Trina Moitra, is the role communities have on retention metrics – maybe something for all of us to think about.
Read the full article
👓 Testing insights from the Global Experimentation Lead at Specsavers
Ben Labay talks to Melanie Kyrklund, responsible for orchestrating experimentation across 10 countries worldwide, each with a unique local market.
With 11 years working in experimentation, she shares what she's learned; from the importance of socializing the testing process over test results, to taking issue with reporting on revenue uplifts. More on that discussion further in this newsletter...but for now, catch the full interview.
People & Skills
😇 Ethical Optimization
Great storytelling in this ABC article, delving into the psychological impact that dark patterns in video games are having on people. The techniques range from some you'll be familiar with from persuasion psychology to some extensive trickery to drive microtransactions; "some games can scrape your social media data and work out, for example, which sports team you support, and then offer you items in the team’s colors."
It made me think about the importance of ethical optimization. When you are in the weeds it can be hard to have that barometer of what's OK and what's overstepping it.
So, here's a good resource to keep you honest and make you aware of what to avoid. If not you might end up getting reported to the Dark Patterns Tip Line a site that shames those who use such patterns.
👀 Brand-side job opportunities
Never a bad idea to keep one eye on the market, to see what specific skills are in high demand. Here are a few interesting roles that have been posted in the last couple of weeks.
- Lead, Experimentation at Macy's (New York, USA)
- Sr. Product Manager, Experimentation at HBO Max (Seattle, USA)
- Group CRO Analyst at Specsavers (Remote, UK)
- Growth Marketer at PayPal (London, UK)
- Head of Product Analytics at Naked Wines (London, UK)
- Optimization Manager at Tesco (London, UK)
- Optimization Manager at Huel (Tring, UK)
Process & Methodology
🧳 Hypothesis Kit
Craig Sullivan has released version 4 of his Hypothesis Kit–the output of eight years iterative thinking on the topic of how best to form a hypothesis. There's some great input and examples from Booking.com too.
While it may appear simple, the hypothesis is one of the most important aspects of the testing process; ill-informed, poorly theorized ones can lead to little wins or learnings and wasted traffic.
The Hypothesis Kit is a sure-fire way to consistently write valid hypotheses. But also a great way to prevent well-meaning "we have to test this" HiPPO-sourced ideas–if everyone has to put their idea into this structure it means everyone has to bring data to the table.
Data & Tools
💰 Can you tie financial gains to experimentation?
David Mannheim has been posting thought-provoking questions about the need and expectation to provide an accurate ROI for experiments, and whether or not we can, or even should measure it.
While most businesses will have some form of revenue attribution calculations baked into their experimentation processes David argues “It is difficult, if not impossible, to provide an accurate ROI for experimentation." But it is still something we have to do to help stakeholders understand the value of experimentation until our practice matures further.
Why can't we attribute an accurate uplift in revenue? A test result is not a transferable or forward-looking statistic, meaning traffic and external changes will shift the baseline, making forecasts or estimates unreliable.
The conversation continues in a live debate: my colleague, Ben Labay will be joining David, Craig Sullivan, Chad Sanderson (Convoy), and Collin Crowell (Kameleoon), to hopefully come to a consensus on the best way to measure the value of experimentation. Save your seat.
About Briefly Experimental
We want Briefly Experimental to become something you genuinely look forward to arriving in your mailbox. A source of the most interesting and thought-provoking articles related to experimentation program management.
But in order to do that we need your feedback. So send your thoughts and feelings to katie@speero.com.