Experimentation / CRO

Experimentation and Testing Programs acknowledge that the future is uncertain. These programs focus on getting better data to product and marketing teams to make better decisions.

Research & Strategy

We believe that research is an integral part of experimentation. Our research projects aim to identify optimization opportunities by uncovering what really matters to your website users and customers.

Data and Analytics

90% of the analytics setups we’ve seen are critically flawed. Our data analytics audit services give you the confidence to make better decisions with data you can trust.

Delivering 5x growth for Codecademy with strategic testing

The Challenge

Codecademy is an interactive platform that offers coding classes in 12 programming languages to over 50M learners across 190 countries.

Operating on a subscription model with a massive user base, Codecademy realized that consistent conversion uplifts could earn millions in additional revenue. But their experimentation program lacked a rigorous process, and many tests were inconsequential changes driven by opinion or what their competitors were doing. There was also a lack of understanding around the key growth levers, meaning that time and resources were spent testing elements that didn’t move the needle.

Codecademy had the potential to drive significant growth and gain a much better understanding of its users through experimentation, but they needed help.

The Solution

The first step was to identify the levers responsible for growth. As Codecademy had a strong brand and top-of-funnel traffic, we focused on growth levers at the bottom of the funnel, such as pricing, checkout, plan mix, and the trial model.

Once the growth levers were identified, we conducted a range of research to understand users' perceptions and behavior throughout the experience. This helped us to identify frustrations, uncertainties, and doubts impacting conversion rates.

With a ton of research-informed hypotheses and a prioritization framework that factored in the impact of growth levers, we moved Codecademy from ad-hoc spaghetti testing to something altogether more strategic.

Experimenting with price and plan mix is complex because they impact the business, product, customer experience, and multiple internal teams such as marketing, finance, customer support, and sales. This complexity often causes executives to call off such experiments. But by using the Speero test workflow, we were able to show the rigorous nature of our process and build confidence to run these complex tests.

“We knew we should be shipping a large volume of experiments but we didn’t know how. The first few experiments we tried missed the mark. But then we started working with Speero who came in and helped us with tooling, research, statistics, and the processes needed to run experimentation at scale.”

Dan Layfield, Growth Product Manager at Codecademy

The Results

Overall our experimentation program yielded 5x growth for Codecademy by building confidence in the testing program, enabling us to run strategic tests on the key growth levers.

We helped increase the number of customers who chose an annual plan by 75% and reduced churn. We achieved this through user research and strategic iterative testing. We found the trial model automatically enrolled everyone who signed up for the free plan onto a Pro version. This meant users couldn’t differentiate between the paid and free features and, therefore, didn’t understand why they should pay for Pro. We ran tests in three iterations—each acting as a stage gate requiring validation before we tested the next version, before finally ending up with the current solution.

Codecademy was able to understand its users better and improve its marketing and product development based on learnings from messaging and proposition tests which were cataloged in a knowledge base and shared with the wider business.

“Through the optimization program, we increased the number of customers who chose our annual plans by 75%, which is a massive deal for a subscription business. With our annual plans, users stay committed to learning for longer, so it also reduces churn. My favorite part about working with Speero is learning. It comes from their team - who sit across areas such as tooling, methodology, and they have ideas that we just wouldn’t be exposed to otherwise.”

Dan Layfield, Growth Product Manager at Codecademy

Key Takeaways


A rigorous experimentation process is needed to move from spaghetti testing to strategic experimentation. research generates $2M in revenue

Use a research-based strategic roadmap to plan and communicate with your team on objectives and key results, helping to build confidence and collaboration with the wider business. Download the strategic testing roadmap template.


Successful experimentation programs focus on complex business critical tests and smaller iterative tests.

To help you balance test velocity vs. complexity, check out this model.


To move the needle, hypotheses must be based on research, not opinion. 

Check out the Research XL blueprint to identify the types of research you need to conduct to form a solid hypothesis.

Let’s kick things off.
Tell us what you’re looking for

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.