Full-service experimentation program

Experimentation should be an ongoing process core to your business, ideally used across teams and departments. Our full-service experimentation programs are a 12-month commitment over which time we not only help you run tests but help to embed the function within your business. 

This question is best answered by understanding how much it’s costing your business while you aren’t validating ideas. The costs come from the time and resources spent working to implement ideas that at best have no impact, or at worse negatively impact your business metrics. 
‍
Additionally, there’s an opportunity cost. While you aren’t experimenting it’s likely your business is more cautious to try new ideas, and therefore the opportunity cost arises from not finding new ways to operate online. Your website traffic cannot be retroactively used to validate ideas so the sooner you start the more ideas you can validate and uplifts you can gain. 
‍
While there are many ways you can experiment and validate ideas that don’t require website traffic, our website experimentation program primarily uses A/B testing. Therefore the best time to start is once your business has the necessary amount of traffic (100,000+ monthly unique users) to run statistically valid tests.

The answer to this depends on the types of tests you are running and the focus of your experimentation program. If you are running basic UX tests you will likely see uplifts in conversion rates. However, you may find you reach the “local maximum” fairly quickly depending on how optimized your website is. 
‍
If you are running strategic tests which are tied to business metrics you are likely to see larger uplifts that go beyond your local maximum, that impact areas such as profit and retention. Alongside validating business ideas and learning about your customers or users.

Ideally, all of the time in an ongoing fashion. The velocity of tests you can run depends on several factors, such as;
‍
● The volume of traffic you have to your website.
● Any blackout periods your business may impose where testing must be paused.
● Any technical work that may impact your ability to test. 
● The resources you have available. 
‍
Mature experimentation programs look to utilize as much traffic as possible, aided by refined processes and automations. However, the types of tests you are running and the journeys they impact need to be kept in mind to avoid issues with conflicting experiences for the user and mudding your results. Setting up swimlanes for your testing roadmap can help orchestrate high-velocity experimentation programs.

Experimentation culture, sometimes also called a “test and learn” culture refers to the mindset and subsequent practices within a business. Businesses who have a good experimentation culture; 

● Use data to inform decisions.
● View failed experiments as an opportunity to learn rather than “a failure.” 
● Make everyone feel safe to call out decisions that are solely based on opinions especially from HiPPOs (Highest paid persons opinion.)
● Have teams across the business involved in running experiments.
● Have leaders who frequently and repeatedly champion the experimentation mindset.
● Dedicate the necessary resources to support experimentation functions.