‍

‍Welcome to Briefly Experimental

This edition was written by Annika Thompson, Director of Client Services at Speero.
‍
Every two weeks we'll deliver the best experimentation content and commentary, curated by a member of the Speero team. We'll break things down into the four key pillars needed for any successful experimentation program.

Edition 10, June

‍

Strategy & Culture

‍

🏁   Prerequisites for an experimentation program

Many businesses start experimentation programs with unrealistic expectations and a narrow focus tied to conversion rate "wins" while undervaluing the broader and overall larger benefits to the business.

These broader benefits include establishing a process that drives continual learnings, testing strategic business initiatives, risk mitigation, innovation, and improving the overall customer experience and retention. But in order to operate in this way, there's a couple of prerequisites needed. 

Ben Labay recently started to develop a list of prerequisites needed to get an experimentation program on the right tracks: 

1. Trust in the system - statistical/data integrity
2. Trust in the team - safety, and vulnerability (to change)
3. Trust in the goal - strategic purpose (as a motivator)

Meanwhile, Alex Birkett also wrote about what he sees as the prerequisite for experimentation. You'll see there's some cross over in the two ideas;

1. Trustworthy Data
2. Human Resources
3. Leadership and Strategic Alignment
4. Experimentation Technology
5. Education and Cultural Buy-In

It led to a lot of debate over what else should be included (or removed) from the list. Jump into the comments section on LinkedIn to add your own thoughts.

‍

‍

Process & Methodology

‍‍

👗 Testing Insights from H&M

Prioritization of hypotheses is always a hot topic. And it's fascinating to see how different businesses approach it.

In the latest episode of Testing Insights, Speero MD, Ben Labay sat down with Matthias Mandiau, Conversion Optimizer, to find out how they approach prioritization at H&M.

Matthias shared how they use PI's or "product increments" to set objectives based on business goals and user friction points and how these play a role in how they set the priority for different experiments. 

Watch the latest episode to hear how Matthias and his team are also creating a searchable central repository of test results and using weekly innovation meetings to get alignment across teams. 

🗑️  How to avoid "trash pop psychology"

Google “Psychologyprinciples for CRO” and you get 7 million results. I get it. Psychology makes for interesting reading and having a list of rules to follow makes life easier.But we’ve written before about why you shouldn’t blindly apply best practices.

Craig Sullivan illustrated this brilliantly, with the story of the paradox of choice, the original “jam choice” study giving way to the principle that too much choice is bad. But Benjamin Scheibehenne, a psychologist at the University of Basel struggled to replicate the original findings. Instead, he found that choice didn’t appear to make an impact on decision making.

So what’s going on? It’s not that the paradox of choice doesn’t exist, it's that it all depends on context. Factors such as the type of product, product information, price and price-presentation, the user's motivation and mindset, and functionality to refine choices, all impact whether this principle has an effect on decision making.

Remove the temptation to create tests that are derived from “let’s test this best practice” by ensuring you have a hypothesis development and prioritization process in place.

Hypotheses and test treatments need to be based on your specific context, which you gain through research and data. Once you understand that, then you can consider which psychology principle might be worth applying as test ideas to solve specific user problems.

‍

People & Skills
‍

‍☠️   Collaboration is killing work

It will likely come as no surprise; “Collaborative work — time spent on email, IM, phone, and video calls — has risen 50% or more over the past decade to consume 85% or more of most people’s work weeks, and the Covid-19 pandemic caused this figure to take another sharp upward tick.”

The numbers are jaw-dropping. It begs the question, how is anyone managing to get work done?

This HBR article references some interesting study findings identifying three main problems when it comes to “collaboration” taking over our working days; volume, structure, and personal motivations. The personal motivation aspect is a really interesting factor, which often gets less focus (if any) compared with trying to change tool usage or processes. 

As an agency, we’re trying out a few ways to tackle this. We’re recommending colleagues ring-fence deep work time using Slack statuses to pause notification.  We’re running monthly “learning time” –a full day dedicated to deep work with short check-ins at a cadence that allows people to “get in the zone.” But we still have some way to go on helping combat our own personal motivations for over collaboration. 

👀   Job opportunities 

Here are a few interesting roles that have been posted in the past week. 

• Product Manager - Experimentation & AB Testing at Lowe's (North Carolina, US) 
• VP of eCommerce Experience at Nutrafol (Remote, US) 
• Senior CRO Manager at ZoomInfo (Massachusetts, US)
• Experimentation Insights Director at Nike (Oregon, US) 
• Global Manager Conversion Rate Optimization at Foot Locker (Utrecht, Netherlands) 
• Conversion Rate Optimisation Manager at Bark.com (London, UK)
• Website Experience Manager at Ooni Pizza Ovens (Scotland, UK)
• Experimentation Lead at Marks & Spencer (London, UK) 
• Senior CRO Analyst at The AA (London, UK) 

Data & Tools

‍

👇  5 Communities where you can find a data science mentor 

Experimentation programs need someone with a solid understanding of statistics and the ability to dig around in data to find opportunities as well as making sense of test results. But these are hard skills to learn (and recruit for.)

Whether you’ve hired a dedicated analyst or someone on your team is looking to up-skill in this area, finding a good mentor can be worth their weight in gold especially if you only have one specialist on your team.

Towards Data Science has put together a list of five communities where you can find a mentor to help provide guidance and support for budding data analysts.

‍