‍Welcome to Briefly Experimental

This edition was written by Ben Labay, Managing Director at Speero.
‍
Every two weeks we'll deliver the best experimentation content and commentary, curated by a member of the Speero team. We'll break things down into the four key pillars needed for any successful experimentation program.

Edition 12, August

‍

Strategy & Culture

‍

🤔  Experimentation as an entrepreneurial strategy?

A cool (newly revised) paper by Koning et al., out of Harvard Business School studied startup performance of those who adopted A/B testing. Check this out from the abstract:

“We find that while relatively few firms adopt A/B testing, among those that do, performance improves by 30% to 100% after a year of use. We then argue that this substantial effect and relatively low adoption rate arises because startups do not only test one-off incremental changes but also use A/B testing as part of a broader strategy of experimentation.”

TL;DR conclusions:

1. Businesses that experiment are more likely to succeed.
2. Experimentation is the preferred framework for decision-making in young businesses.
3. We should reconsider emergent versus intentional approaches to strategy in the context of entrepreneurship. The researchers found that experimentation helps drive both valuable incremental changes and the development of significant product improvements.  

That leads right into the importance of process.

‍

‍

Process & Methodology‍‍

‍

‍⚖️  Experimentation scale comes from program management

Scaling experimentation isn’t about the testing tool. It’s about program management.

DoorDash increased its testing velocity by 1000% in 3 years by focusing on the program itself. What they focused on is covered in this great piece by Sifeng Lin and Yixin Tang.

To get scale, DoorDash focused on:

• Metric standardization (goals, guardrails, and diagnostic metric taxonomy.)
• Automation of test analysis, especially for low power (low sample size) testing.
• Standardized rituals with testing phase gates.

🔎  Evaluating Microsoft Teams across build releases

“A/B testing is like a unit-testing tool. In practice, A/B testing is rarely used for comparison between whole builds.”

Check out this wonderfully elegant article on how Wen et al. are tweaking the standard structure of A/B tests for integration testing. But, by building a new version for testing, you have two ‘effects’ on the test that the Microsoft team observed:

Penetration difference: A/B test variations can have the same traffic but the composition is different, violating assumptions of normalcy.

Update effect: for software A/B testing, when updating to a new version you get benefits of cleared cache, so a performance difference.

Doing an A/A/B test allows the analysis to handle these issues. Elegant.

‍

🏦  Testing Insights from Ally Finance

On this episode of Testing Insights, I ask Patrick Buffum, Senior Digital Optimization Analyst at Ally Financial three questions:

1. How do you filter out and prioritize quality experiments?

2. How strict and incremental should your hypothesis be?

3. How closely should the testing strategy be tied to the larger business strategy?

Watch the episode to hear why you should use diverse team members to hash out ideas and avoid being overly scientific.

‍

People & Skills

‍

‍👮  Who polices the police? 

Who governs your experimentation program?
I’ve been working with Manuel Da Costa on program audits recently, and it’s truly fascinating work. I love the voice of Manuel on this topic. He is doing great work and promoting the point of view that leadership should be involved from the start of any experimentation program setting up roles to properly govern them.

‍
Paraphrased from Manuels’ recent post, it’s important to:

1. Get senior management involvement and a board member that champions true experimentation.
2. Don't just fill teams with technicians. You need to have orchestrators who manage the programs and ambassadors who engage and evangelize.
3. Build governance and review processes in place.  
4. Unify team objectives across UX, CRO, SEO, etc. When team objectives are aligned, collaboration is easier, even if they do different work.

👀   Job opportunities 

Here are a few interesting roles that have been posted in the past week. 

• Experimentation Manager at Amazon (New York, US)
• Senior CRO Strategist at Corra (Remote, US)
• CRO Manager at Raegan Hill Group (Remote, US)
• CRO Lead at CV Screen Ltd (Cardiff, Wales, UK)
• CRO Manager at Eurostar (London, UK)
• CRO Content Optimisation Director at PlayStation (London, UK)
• CRO Manager at MRJ Recruitment (North Yorkshire, UK)
• Growth Manager at Conversionista! (Stockholm, Sweden)

Data & Tools

‍‍

🧠  Mapping out the data you need 

Booking.com has conversion rate as its key driver metric within experimentation programs because it's too hard to set up LTV. I think this is going to change more and more. Data sources, data pipes, data storage, data reporting. Do you have all this figured out?

With this ‘data map’ handled, you can answer these questions within a test:

• What subscription products are associated with the highest LTV?
• What is the relationship between subscription price and churn rate?
• What are the estimated upcoming orders for the next 30 days?
• What are the refund rates per channel?
• What channels have the highest LTV?

This isn’t BI. BI is dead. The new term is Data Map. I coined it ;) Here’s a great piece by Benn Stancil on this topic of why connotations around terms such as BI hold us back.  

‍

📈    Are you using GA4?

Google’s new platform is ready to collect useful insights and the sooner you start gathering data, the more you can enjoy the advantages of Google’s Machine Learning expertise to enhance your datasets.
Since some features of Universal Analytics are still missing, I suggest you keep your existing analytics properties in place for now, while you start familiarizing yourself with GA4.

My colleague Mate Varju wrote a thorough overview of all of the features of GA4 (as well as what’s still missing). But the custom reporting options, machine learning features, and the promise of eventually having an all-in-one tool to tackle cross-device, cross-platform fragmentation in the ever-changing world of data laws and regulations, more than makes up for the current missing functionalities and transition headaches.

Our data and technology team has already started working on setting up and configuring GA4 so if you need a helping hand get in touch.