The State of Experimentation Programs 2021


This is the first year of our Experimentation Program Benchmarking Report, something we hope to do every year to show how experimentation programs are developing, and helping us to advise businesses at different stages of maturity on what they can do to improve their experimentation practices.

What we found

We found a surprising number of relatively easy-to-implement areas that most businesses are not doing but should be, to accelerate their programs. 

We’ve also found a number of areas that only the highest maturity level businesses have in place. Strong indicators that if you want to increase the effectiveness of your testing program these should be your focus areas. 

We’ll cover three of the headline findings from the report. But if you’d like to get the full benchmark report you can get your free copy here. Our experimentation program maturity audit is also still available for you to assess your own program.

Research Methodology

A total of 210 respondents from brand-side roles answered the Experimentation Program Maturity Audit. We solicited responses via our company newsletters, website, LinkedIn advertising, and social media channels, between September 2020 to January 2021. 

The audit asked respondents to answer 33 questions around four key areas needed for a successful experimentation program;

Speero's four key areas needed for a successful experimentation program
Four key areas needed in experimentation programs


The questions were either presented using a Likert 0-10 scale with bipolar adjectives “don’t agree” and “strongly agree,” or as multiple-choice questions. 

In order to apply a narrative to the dataset, we employed a similar weighing to our Likert scale as used in Net Promoter Score. Scores were grouped by 0-6, 7-8, and 9-10. We attributed statements to these bracketed scores: 

  • 0-6: Disagree with the statement
  • 7-8: Somewhat agree with the statement 
  • 9-10: Strongly agree with the statement

Maturity Levels in Experimentation Programs 

Each respondent received an “overall experimentation maturity score” based on the answers they provided. The scores correspond to different levels of maturity as follows:

Speero Maturity Levels in Experimentation Programs 
Maturity Levels in Experimentation Programs 


Beginner–an overall score of 0-20%

Businesses at the start of their experimentation journey. Few of the fundamental building blocks needed to run an effective experimentation program are yet to be implemented. 

Aspiring–an overall score of 21-40%

Businesses that have established some of the important elements needed in preparation for running an effective experimentation program. These businesses typically have many internal hurdles to overcome and practices to implement in order to run a successful experimentation program. 

Progressive–an overall score of 41-60%

Characterized by businesses that are starting to recognize the importance of insight-driven experiments and the need to improve their processes to increase the performance of their work. They have the necessary foundational elements in place to run a basic experimentation program. 

Strategic–an overall score of 61-80%

Businesses that have most of the foundational and some advanced practices in place, employing a strategic approach to experimentation. They are likely to have wider company buy-in for experimentation as a core business growth driver due to the results from their work. 

Transformative–an overall score of 81-100%

These businesses are the industry elite. They are outperforming their competition through a well-oiled experimentation program that is consistently delivering results. 

Three key takeaways from the report

1. Too many businesses are making decisions without considering customer data and insights

This question really gets down to the nitty-gritty. If you want to run an experimentation program that’s effective you need your business to recognize that decisions should be informed by data.

If not, your test might prove a certain hypothesis but a HiPPO (Highest Paid Person Opinion) can swoop in and pull rank over your idea, based on their own experience (gut) or agenda, despite irrefutable evidence. And it seems like this might be the reality for many businesses–we found that over half (54%) of respondents disagree that business decisions always consider customer data or insights–with only 20% strongly agreeing. 


How many businesses consider customer data and insights in decision making? Speero Research

Test ideas also need to be based on data and insights otherwise your win rates will be low and you’ll jeopardize any support for the program. But collecting customer data isn’t always easy, turning that data into insights is even more difficult. Not to mention, the seemingly continuous work of having to add new data sources/data points into a structure where it can be analyzed and made actionable. All of which takes time, money, and skilled resources. 

This is likely the reason behind why we’re seeing the majority of businesses before the ‘transformative’ stage either ‘disagreeing’ or only ‘somewhat agreeing’ that business decisions across their company always consider customer data and insights. 

What can you do about it? 

Experimentation can help prove or disprove business ideas you want to make. But in order to do that, you need a sound hypothesis development process, usable, trustable data, and customer research. It’s a chicken and egg situation for many businesses and requires leadership to step in to make the necessary investments and cultural changes to support data-driven decision-making.  

2. Sharing experimentation learnings across your company is something that most companies can benefit from

How many businesses share their experimentation learnings - Speero Research


Overall, only 14% of respondents strongly agreed that they consistently shared experimentation learnings across their company. When we look at the level of agreement with this statement by maturity level, we see that 67% of ‘transformative’ businesses “strongly agree” they do this in their business. 

While on the surface this appears to be an easy-to-do task, it’s not really the case. What’s needed is the right message, in the right format, to the right people, at the right time. To make this work you’ll want to establish a set of rituals and methods that encourage not just the consumption of the information but engagement with your work. 

Sharing test learnings as well as encouraging engagement in experimentation is an area that even beginner-level businesses, can and should do. While this task is more than just sharing the information, rather, it’s about getting engagement and buy-in, the barrier to starting such initiatives is low. 

What can you do about it? 

At Speero we tend to create client-branded monthly newsletters for our point-of-contacts to share throughout their business, which highlights the tests running, the outcome/results, and learnings. 

But it can be even simpler. You can post updates in a work slack channel and use polls to encourage engagement on which variant people think will win. We’ve even done internal testing to track open rates and click-through rates of internal communications such as those newsletters to optimize how we communicate the information. 

Given the low barrier to starting such initiatives but the high impact, this is a no-brainer for most of the companies in our survey. Getting a win is half the battle. Communicating that win to encourage engagement, ideas, and team alignment will truly spur innovation and a progressive experimentation culture.

3. Only 19% of businesses surveyed are experimenting with customer loyalty despite the retention economy

Most popular areas to run experiments - Speero Research

The most common test areas are those which are more tactical in nature (first 3 columns from the left.) Such experiments are;

  • Easier to set up. 
  • Usually don’t need buy-in from other stakeholders.

It’s very easy to get started by testing color, copy, and images, and it’s a good way to get into the “testing mindset.” These types of tests are normally doable without a developer, using visual editors within the testing tools themselves. 

But testing new features, products, and pricing, requires a more strategic approach as well as development resources, potentially server-side testing capabilities, and other team’s involvement. These strategic tests are where businesses find innovation and usually the biggest wins, and thus should be the goal of any testing program. 

What’s interesting is that only 19% of those surveyed are testing around customer loyalty or referrals and 22% around customer support. These need to be focused on, as we find ourselves living through the retention economy. 

For too long businesses (and the optimization industry) have been pressured into short-term strategies and quick win tactics that drive revenue, rather than working to create long-term growth.

But the cost of acquiring a new customer is 5-25 times more expensive than retaining existing ones. Therefore the winners of the retention economy will be those who measure, test, and improve customer retention, understanding that a 5% increase in customer retention leads to 25% - 95% more profit

What can you do about it? 

Our research shows that there is a massive opportunity for businesses to get ahead of their competitors by focusing testing efforts around increasing retention and customer lifetime value. To develop hypothesis around these topics, conduct customer research to identify issues and opportunities and then prioritize your hypotheses for testing.

Get the Experimentation Program Benchmark Report 2021

To get the full report with all 14 findings and advice on what you can do to improve your own experimentation program based on the research, download your free copy here. 

Related Posts

Who's currently reading Briefly Experimental?