How Gympass started and scaled experimentation across its business in just six months

Gympass—a complete corporate wellbeing platform. Companies around the world rely on us to support their employees wellbeing, health, and happiness. Founded in 2012, with HQ in NYC, we have a global team in 11 offices around the world.

A couple of years ago, Gympass had no experimentation program. My VP and I (Shagun Aulakh) decided to change this. During the process, we went from zero experiments a year to having a mature experimentation culture with the help of Speero. This is the story of how it happened.

The Foundation Was Already There

The foundation of our experimentation program was there even before I started. My VP already had experience and ethos of experimentation. During one of our first meetings, she told me that there was zero experimentation happening in marketing. She wanted to figure out how do we build an experimentation program from scratch.

The end goal was clear—we need to make better, data-driven decisions. The primary foundation for this would be experimentation. However, with this came another problem. Gympass had no experience with experimentation. Yes, we had a general understanding of what experimentation entails.

• But how do we run experimentation programmatically?
• How do we drive real strategic insights?
• How do we design experiments? 

Our team didn’t have these foundations. This is where I came in because of my experimentation background. My role was be to be the ‘ground zero’ from which experimentation would spring up across the organization. My vision was to make experimentation the DNA of Gympass, and not just something that stays in the marketing department. Make everyone not just experiment, but have that experimentation mindset.

How to Start Experiments from Scratch

I needed to build an experimentation program at Gympass from scratch. First in the marketing team, but then across several departments. My first move was to analyze our toolset:

• What tools are available? 
• Are they the right tools for the job? 
• What team members we have? 
• How much they know the tools?

In the beginning, we were only using one tool—UA. Even that wasn’t set up correctly. Analytics were a mess. No one really knew how to use it. So we went through a period of selecting a new vendor (we went with Optimizely).

After this was done, it was time for my second move — education. So there were a lot of stakeholder meetings and coaching them on how to experiment. But I didn’t start with teaching about experiments. At first, I asked stakeholders important questions:

• What are you trying to achieve in your area of business?
• What are your challenges?
• What are your questions?
• Which metrics are you trying to move?

I combined answering these questions with coaching them on how they can craft their experiments.

Helpful Blueprints:

— XOS Tool Stack: your overview of most common software tools for experimentation, including examples of specific tools in each category. Use XOS Tool Stack Blueprint to assess the required software tools for an experimentation program and to budget and estimate costs.

— Cadence for experimentation meetings: Experimentation is a process. You need meetings to coordinate, align, and communicate. This framework lets you pair all of this together as it provides questions to ask during weekly, monthly, and quarterly meetings.

— How to engage community and create culture around experimentation: create a culture of experimentation in your company, and make it accessible and enjoyable. Train and educate, increase engagement and buy-in.

Our Grail — Test Velocity

In the beginning, my main metrics was ‘just get an experiment started’. It took us four months to reach it. And the experiment wasn’t even complex! As we continued to grow, our focus shifted to test velocity—the number of experiments we could run. 

We aimed to increase the quantity of experiments across different teams and functions. This is where Speero comes onto the stage. They helped us:

• Establish and run the experimentation program.
• Drive strategic insights and a testing roadmap through Research XL.
• Create standards for reporting and knowledge sharing to align teams across the business. 

We emphasized adopting and using these standardized templates for better governance and shared understanding. Everything—from test design to analysis—was templated. This way, all teams spoke the same language when it came to experimentation.

In fact, we tracked how much different stakeholders adopted and used those templates. At first, it was 10%. But soon it was 70%-80%. This enabled us to access the level of experimentation adoption across the business.

Speero also helped us establish internal forums and monthly meetings open to any team across the business. I tracked the number of participants in those meetings to ensure we engaged with diverse stakeholders from marketing, product, analytics, and engineering.

Monthly experimentation share-out meetings became a platform for cross-functional learning and collaboration, fostering a sense of healthy competition. The product team would join and listen to how the marketing team was experimenting. Then they would say, “the marketing team is ramping up experiments, how can we reach them?

This lead me to realize we need a more nuanced testing velocity. Not just the aggregate, but the specific team velocities. How much did acquisition experiment? How much did marketing and product teams experiment? I wanted to drive the culture of experimentation. Embed it into Gympass DNA. And this wasn’t possible if the end result was “yea, we ran fifty experiments, but only in the product department”.

Speero helped us gain deeper insights by breaking down the experiments by type, location, and impact on the user experience. Is this an iterative test, substantial test, or a disruptive test? This allowed us to evaluate the sophistication of our experimentation efforts and target areas for improvement. Our journey involved constant adaptation, evolving metrics, and a keen focus on building a culture that embraced experimentation.

Today, teams don’t require hand holding. They’re running their own workshops. They are independent when it comes to more complex tests and questions.

Helpful Blueprints:

— Solution Spectrum: Tag your tests based on the disruptive scale (Iterative, Substantial, Disruptive) so you can get more disruptive in your tests and measure your test portfolio balance. Mature programs are often a balanced combination of big bets focused on learning and little tweaks concentrated on margin gains and optimization.

— ResearchXL (RXL): Identify the biggest barriers to conversion and vital customer issues in your UX. RXL can be a strong foundation of your testing program, letting you design tests based on data and user research, instead of HIPPO and guesses. Your alternative to randomized testing.

— How to balance test velocity vs complexity?: Benchmark the state and progress of your experimentation program. Monitor its health and use this blueprint as a coordination and alignmnet tool for your team.

The Four Pillars of Success

The success of Gympass experimentation program can be attributed to four key pillars: 

• tools and technology
• people and skills
• culture and strategy
• process and governance. 

By focusing on each of these pillars, the program ensured that the right tools were in place, team members were trained and onboarded effectively, a culture of experimentation was fostered, and robust processes were established.