For many growth marketers, “move fast and break things” has become a mantra. The quote can be attributed to Mark Zuckerberg, who said “unless you are breaking stuff, you are not moving fast enough.”
So instead of debating methods of growth and rolling them out to find out if they have the impact you’d hope for, high-velocity testing can figure out what works, and doesn’t, fast. And many tech giants (Facebook, Google, Airbnb, Twitter, etc.) credit their success to high-velocity testing. We have seen it ourselves when helping businesses such as Mongo DB and Miro go from 0-100 test velocity per year in the space of a few months. Here’s what we learned and what you need to know to do the same in your business.
What is high velocity testing?
High-velocity testing is where you maximize the traffic/audience you have to test on, running as many experiments as you can. Because the more tests you run, the more ideas you validate, the more you learn about your audience and what works (and doesn’t) for them, and thus the quicker you can grow.
While high velocity testing sounds like a no-brainer, it seems few businesses are able to execute it. In our State of Conversion Optimization 2020 report, we discovered that the average company only runs 1-2 tests per month. And the companies who performed 20+ tests each month were in the minority–only 9.5% of those we surveyed. So high-frequency testing isn't exactly common.
What’s holding companies back from high velocity testing?
- Low budgets for optimization which reduce the resources and tools needed - we found that CRO teams rarely have their own dedicated budget.
- Nearly a quarter of businesses revealed that no one had the primary job of optimization. So there’s a lack of accountability, leading to a lack of focus on testing, let alone high-velocity testing.
So while there are internal barriers, let’s explore why you can do to work towards high-velocity testing with some good reasons to push things forward within your business.
How can you do high velocity testing?
To carry out high velocity testing, you need to execute your test ideas at a rapid rate, across the whole company. Here are the main ingredients;
1. Constant ideation
You’re going to need an endless supply of test ideas to feed your testing program. Even if you have a dedicated team for this, it’s difficult to keep up without stagnating. So, it’s essential that you have a process in place to keep the ideas coming.
To give your ideation some structure, you can use the AARRR startup metrics model developed by Dave Mcclure to consider each of the stages your customer past through. Alternatively, journey mapping your entire experience can give you a more granular guide on where to focus. It’s important not to forget that offline and internal teams can also use experimentation to aid growth, from customer care to logistics processes.
Once you know where to focus the best way to generate ideas is to conduct cx research. Our ResearchXL model gives you some of the research methods to use to generate hypotheses. It’s not an exhaustive list by any means but should stand you in good stead for getting started.
2. Hypothesis prioritization
We’ve previously written about how you can objectively decide which ideas are worth testing. Without a prioritization process, it’s easy to fall into the trap of running ineffective tests just for the sake of being ‘high velocity’. But if you aren’t getting learnings or results, your program won’t be worth running.
There are multiple prioritization models you choose from to rank your ideas. This includes a framework inspired by Wayne Chaneski’s ICE framework:
- Impact - What is the potential benefit to the company?
- Cost - How much will the test cost to implement?
- Effort - What resources and time are needed for this test?
You can only score each component either 1 or 2, depending on whether the answer is high or low. Once you've added up the total for each component, you can use the total figure to rank against other ideas.
We're fans of this model, as it’s a simple binary approach. But the limited set of criteria leads to having a lot of ideas all ranked the same, especially when you have a lot of potential ideas. That’s why we created our own prioritization model called PXL which you can download and use for free. It attempts to remove as much subjectivity and personal bias while ensuring that everyone brings data to the conversation. The benefits of the PXL models are that it:
- Makes rating the “potential impact” and “ease of implementation” objective.
- Helps foster a data-informed culture.
- Has a wider set of criteria (which can be customized to your business) to provide a more nuanced ranking of ideas.
3. An efficient test process
In order to carry out high velocity testing, you’ll need to optimize the experimentation process itself. You can have a bunch of brilliant ideas lined up and ready to go, but if you don’t have an efficient process to test and report on them they will sit there gathering dust.
To help you optimize your own experimentation process we created an audit so you can assess your maturity in four important areas;
- Strategy and culture
- Tools and data
- People and skills
- Process and methodology
You can access the experimentation program maturity audit here.
From our recent industry survey, it seems one of the first stumbling blocks for many businesses is not having any dedicated resources. You need to have at least one dedicated person who takes responsibility for the experimentation process. But a team with a range of skills is more likely required if you want to move into high velocity testing.
If your business is more mature and has a team in place, it’s important to consider how it’s structured to get the most out of the team. We’ve covered the pros and cons of centralized, decentralized, and center of excellence team set-ups before.
Measure test program metrics
Having a focus on the right program metrics can help you optimize your test process by understanding what’s working and where there’s room to improve. Here are the top metrics we recommend you measure:
- Test velocity: How many experiments are you running in a set timeframe?
- Test efficiency: The percentage of experiments with production issues (or the number of days delayed).
- Test quality: The number of impactful learnings you make from your experiments.
- Program Agility: The number of days it takes from hypothesis, to live test, to implementation (when it’s a winning test).
- Win rate & % uplift: It’s important to measure both - lots of wins with tiny uplifts won’t drive a positive ROI so you’ll want to improve both of these metrics for a healthy testing program.
4. Record and share your insights
After tests have run, it's really important to record what you've learned. If you don't, you could end up repeating tests you’ve already run. This is especially true if you have a large experimentation team or are testing across the business.
By maintaining a test database, you'll find it much easier to communicate learnings to your co-workers and quickly find tests you’ve previously run. Having the data and evidence in a centralized place will also help you justify any changes to your bosses or stakeholders if they ask for it.
But don’t only focus on the tests that were successful, focus on the tests that gave you an insight. The test doesn't need to be deemed a commercial 'success' for it to be worth recording. To understand how best to archive your test results, head over to this blog post on the topic.
How can high velocity testing go wrong?
While high velocity testing can accelerate your decision making and growth, there are certain issues that you should prepare for. Here are three common pitfalls:
1. Your business doesn’t have an experimental mindset
For high velocity testing to be successful, you need a culture of experimentation throughout the whole business. To encourage this you can:
Get test ideas from every corner of the business.
Give customer service, sales, product teams, etc. an easy way to submit ideas or insights for you to use. For more mature businesses, helping these teams run their own experiments will help you increase your test velocity too. Josh Aberant said that during his time at Twitter, everyone in the business was encouraged to run an experiment on 1% of their traffic, and they made it a faux pas to turn up to meetings without test results.
Given most businesses don’t have hundreds of millions of daily users, this strategy isn’t applicable to most, but the concept behind it is. To run a high velocity of tests you need a company-wide approach to testing.
Celebrate failed tests as much as successful ones by showing what learnings you made.
Ensure other teams can use those learnings e.g if a website experiment provided insights on which proposition messaging worked and which didn’t, make sure your marketing team is aware of the results too. We use a monthly newsletter that’s sent out company-wide for clients, to help share this information internally.
Gamify the experimentation process.
Laura Borghesi, Senior Director of Growth Marketing at MongoDB told us that she and her team take bets using Slack Polls on the outcome of tests to help encourage engagement throughout the team.
2. Calling tests too soon
Carrying out tests at high speed could lead to complacency. As time and traffic are tight, you might assume the results are ‘cooked’ and call tests too soon. But you shouldn’t stop a test just because the testing tool says it’s significant, just as you shouldn’t keep it running because it’s not significant.
Every test needs to have a predetermined and fixed time frame to run. Setting this in advance and sticking to it will ensure you won't call tests too soon and pollute your results. You can use our A/B test calculator to calculate how many people you need to reach before you decide to call your test.
While calling tests too soon can skew your results, so too can allowing your tests to run for too long. With many people deleting their cookies regularly, users could re-enter your test as a newcomer, so it's generally best to call time on a test within a 4 week period if possible.
3. Quantity overtaking quality
With a high velocity testing approach, it's easy to get caught up in the speed, and neglecting the quality.
This is where prioritization of your ideas comes into play. But equally important is the execution of the test itself. You can answer the same problem in hundreds of different ways. Using established psychology/behavioral, UX, and usability practices can help you produce a better execution of ideas, that are more likely to win. So spend as much time working out what to test as how to test it.
How can I measure the velocity of my testing program?
It comes down to three factors: testing capacity, testing velocity, and testing coverage.
1. Testing capacity
How many tests can you actually run? To work this out simply divide 52 (weeks in the year) by your average test duration. Then, multiply this figure by the number of different pages/funnels/places you can test on at any one time.
2. Testing velocity
Once you have your capacity, you can then track how well you're working towards it by recording how many tests you actually run. To get the most out of this measurement looks at trends over time to see if things are moving in the right direction.
3. Testing coverage
Out of the days that are set aside for testing, how many days are you actually running a test on? You might have some testing blackout periods e.g. over the holidays period for ecommerce businesses. Try to factor these in, to get a realistic figure.
Testing coverage will show you whether you're wasting valuable testing time and traffic. If there’s traffic you're not utilizing, ask yourself why and what you can do in each specific area.
Conclusion
While Zuckerberg was about "moving fast" being the key to growth, he didn’t take into account all the other factors which are needed to make high velocity testing a success. Such as a set process, due diligence, and dedicated resources.
To ensure your high velocity testing program delivers results, you need to make sure you have:
- A constant stream of ideas
- An effective prioritization framework to rank ideas
- Dedicated optimization resources
- Optimization of the test process itself
- A culture of experimentation
You'll also need to measure these important metrics so you can work out how successful your high velocity testing program is:
- Testing capacity - how many tests can you run?
- Testing velocity - how many tests are you carrying out?
- Testing coverage - how many testable days are you running a test?
.svg)
