Welcome to Briefly Experimental
Every two weeks we'll deliver the best experimentation content and commentary, curated by a member of the Speero team. We'll break things down into the four key pillars needed for any successful experimentation program.
This edition was written by Viljo Vabrit, Co-Founder of Speero. If you'd like to geek out with Viljo on any topic mentioned in this newsletter, just hit reply and you'll reach him directly.
Edition 13, October
Strategy & Culture
💯 How advanced is your experimentation program?
Wouldn't it be great if you could systematically evaluate your experimentation efforts? Find the gaps and opportunities across strategy, process, people, and data? Even better, have a document to share internally, to help get buy-in and internal alignment. How about a benchmark to assess your own efforts compared with other businesses' experimentation programs?
Enter, our Experimentation Program Maturity Audit. It's free and will help you evaluate your current performance, measure your gains over time, and allow you to compare your efforts with other businesses.
"Before we did the audit it was already clear that we have a long way to go, but we didn't have enough clarity to decide on what to do next. Doing the audit gave us the map we needed to bring our CRO efforts to the next level in the coming year." - Bas Boland, Vitaminstore
Get your free experimentation program audit
😃 CX relies on employee satisfaction
At the recent Festival of Marketing, leaders from Tui and Co-op came together to discuss their priorities for 2022. This quote sums up some of their discussion;
“Businesses and organizations that have heavily invested in - not just prioritized - employee and partner enablement are leading the way in terms of customer experience. It's that notion that to be customer-centric, you must first be employee-centric”. - Roy Capon
Both leaders had arguments and data that backed up this wisdom. And as we know, an excellent customer experience is key to customer retention and business growth in general.
Two takeaways from the article to put this into practice:
- Adapt the criteria used in your recruitment process to prioritize soft skills, values, and beliefs that align with your core values, rather than experience.
- Make sure your employees see your customer data and feedback because that's how they see the impact of the part they play.
Process & Methodology
🧈 Detect churn risk factors in your ecommerce store
Yes, we're talking churn in ecommerce. It's a hard nut to crack but believe me when I say it's a massive boon to your business if you are able to predict when a customer might not be returning and you can put things in place to encourage them back.
This handy article from Omniconvert covers some good reminders of what to look for in your data to identify when your customers may be about to churn. A couple of important ones to monitor are:
- Responses to customer satisfaction surveys.
- Newly acquired customers who haven’t placed another order within the average days between the first and the second order.
- Loyal customers who haven’t placed another order within the average days between transactions.
- High return rates for specific brands, products, or categories.
People & Skills
♾️ Experimenting on experimentation teams
"The way your team is configured has implications on knowledge distribution, architecture of products, career growth of individuals, effectiveness, and many other aspects." That's why the personalization team at Adevinta has spent the last 6 months testing out five different team configurations.
While they stress there is no "one approach fits all" their process and approach to this problem makes for an interesting read that will hopefully help you reflect on your own team structure.
Read more
🎯 Simple rules that are hard to follow
Speero has been a "hybrid work" company by choice since we started ten years ago. Our people are working from 16 different countries. Being able to recruit the best talent without geographical limitations has been one of our competitive advantages.
For most traditional organizations where employees had to show up to an office every workday, this hybrid way of working has been something forced upon them by the covid crisis.
Several big organizations like Netflix and Facebook are "forcing" people back to the office. I guess it´s because traditional people management consists of micromanaging what looks like a productive workday instead of tracking what is actually being accomplished.
This short read from Tsedal Neeley offers three rules for senior managers to forge a new hybrid path for their organization.
- Trust in employee autonomy: "You have to trust that they are intelligent enough and well-intentioned enough to get work done no matter what it takes," Neeley says.
- Stop counting 'butts in seats'. You don't have to see people working to get results.
- Support work-life wellbeing. "When people are working from home, they are giving up actual real estate in their households to their companies," Neeley says. "The least you can do is care about how they are."
Simple rules, aren't they? Sometimes the simplest things are the hardest to execute.
👀 Job opportunities
Here are a few interesting roles that have been posted in the past week.
- Senior Experimentation Strategist at Speero (Remote, US)
- Experimentation Director at Comcast (Remote, US)
- Web Optimization Manager at Fender (California, US)
- Assistant Manager, Digital Experience Optimization at Adidas (Oregon, US)
- Digital Optimization Specialist at Wiley (Remote, US)
- Experimentation Director at Stitch Fix (Remote, US)
- Digital Optimization Strategist at Widerfunnel (Vancouver, Canada)
- Principal Product Manager- Experimentation at Skyscanner (London, UK)
- Senior Digital Experience and CRO Manager at Wilko (London, UK)
- Conversion Optimisation Manager at The Telegraph (London, UK)
- eCommerce User Experience (UX) Optimisation Manager at L'Oréal (London, UK)
Data & Tools
Data & T
🏆 Wayfair's tool to forecast "delayed rewards"
We've talked before about the importance of understanding longer-term metrics when optimizing an experience. Without understanding these metrics you could be making changes that are positive in the short term but harm the business in the long run.
But, as in many businesses with long sales cycles, this could mean having to wait a while to measure orders and revenue lift (what Wayfair refers to as “delayed rewards.")
So how can you maintain a fast learning pace while also understanding the long-term impact?
Wayfair has attempted to solve this by developing "Demeter" their own data science platform that uses ML models to forecast longer-term KPIs based on customer activity in the short term.
In this article, they discuss the theoretical foundations of the “delayed reward” forecasting problem and explain its use cases for not only testing but for “reward generation." It's a problem many businesses will be looking to tackle, wrangling large data set to make better decisions, so this is well worth a read.
News & Events
News: Changes to the Hotjar basic plan
We knew when Hotjar, the heatmap and customer feedback tool, got bought by Contentsqaure last month it would herald changes. Here's the update we've all been waiting for in terms of what's going to be included, and not, in their free plan going forward. The good news is they are keeping their free plan.
Event: Should you experiment with pricing?
If a customer will pay more for a product or service, then what's wrong with charging a higher price? Enter the curious world of price experimentation, where ethics and logic collide.
To discuss the merits, my colleague Emma Travis joins guests from TalkTalk, Shopify, and Kameleoon in a live session on Wednesday, 10th November.
It's free. Sign up here
Event: CXL 10
People into growth and experimentation are coming together on November 17th in Austin, Texas. Speero co-founder Peep Laja is throwing an event where the main aim is extending your personal network with top names in growth. There will be:
- Lightning talks by industry top thinkers and doers.
- Structured discussions around attendees' specific challenges and goals.
- A series of dinner parties designed to connect like-minded people.
Only a few tickets left. Grab yours here
Event: ConversionJAM
A series of online virtual events starting in November. The site lists four sessions covering changes in customer behavior post-pandemic, cookies and their impact on testing, and how the hybrid working environment is impacting our work.
Tickets are free - get them here
Event: The Conference formerly known as Conversion Hotel
For the 8th year in a row, the Online Optimizers Meetup Group will run (The Conference formerly known as) Conversion Hotel aka Experimentation Island on November 19th.
#CH2021 is a 3-day event on the island of Texel in the Netherlands. Passionate digital optimizers will spend the weekend sharing their knowledge, experiences, and skills in the field of Product, UX, Marketing, and Data.
Live and virtual tickets are available here
Event: CROshop Live
VWO is running "CROshop" a live virtual, and interactive workshop (NOT your typical webinar) aimed at getting you up and running in CRO. There are workshops for North America, Europe, and APAC focusing on different verticals, spanning the next two months.
You can catch Speero's MD, Ben Labay, on the 1st of December in his interactive workshop. Learn how Codecademy strategically aligned their testing program to a central goal and doubled down on that goal to get 5X growth within a year. Ben will use Miro as a collaboration tool to work with participants on their own metric taxonomies for growth, program driver metrics (SaaS specific), and program guardrail metrics.
Find out more about the speakers and dates here. Note: CROshop will not be recorded.