Speero Experimentation Blueprints

What's in a Strategic Testing Roadmap? Blueprint

A strategic testing roadmap Blueprint is the culmination of research and the basis of a great OKR-style action plan for a testing program. The boxes are insights that come from triangulated research data (quantitative and qualitative). Some boxes are strategic, and some are tactical, but overall, it's a punch list. The 'Key Results' part of the OKR.

The objective part is framed as a powerful 'how might we...' question. And this question isn't determined ahead of research, it comes from the research itself, after coding among the insights and finding problem or opportunities patterns. The specific KPIs aren't important, but now you can make the goal SMART. You go through the punch list of insights and watch the needle move on those goal-associated KPIs.

Use Cases:

- Create a research-based strategic roadmap for a testing program.
- Communicate with your team on objectives and key results for a test program.
- Organize tactics and strategies against research and metrics.

What meetings should I have with my experimentation teams? And when? And how?

Cadence for Experimentation Meetings? blueprint helps you answer these questions. Experimentation is a process, so you need meetings to align, coordinate, and communicate. This clever little blueprint lets you pair all of this together by providing the questions you should ask during quarterly, monthly, and weekly meetings.

Use Cases:

- Create systems of feedback with your team
- Effect the culture of your program or org
- Train and educate your team
- Have accountability systems

‍

What are the result vs action options for an AB test?

The result of a test can have little to do with what comes next. So it's important to separate the concepts of 'result' from 'action' so you can independently track how your program is doing related to 'win rate' vs 'action rate'. The goal is not to 'win' with tests, it's to make good decisions that effect change. Results Vs Actions Blueprint lets you report on the test you ran and their impact. 

It lets you develop tags in your tests as results or actions coming from that test. This way, you’ll be able to calculate your win rate and testing agility. 

The true power of this blueprint becomes apparent only down the line. Once you implement it, you can get the portfolio of your testing program. Over time, you’ll see the difference between substantial vs disruptive tests, their results, and actions made from results, all of which become powerful from a program perspective once you start to scale the number of tests.

Use Cases:

- Calculate 'win rates' vs 'action rates'.
- Look at the percentages of iterations vs what was shipped.
- Show how test results aren't connected to outcomes, changes, for a website for example.
- Measure the agility of your experimentation.

‍

How To Balance Test Velocity vs Complexity?

Blueprint is probably one of Speero’s favorite frameworks to benchmark and understand the progress and state of our clients’ experimentation programs. This blueprint tells the story of two opposing forces—Better tests with more impact VS running more tests. This is a never ending battle. You can’t have the cake and eat it too. 

The key here is balance. You should be constantly moving toward the upper right side of the graph—run more tests.  But as you run more tests, you want to slowly build the capability to run more complex tests and to run the more efficiently every time you jump a certain ‘hurdle’ on this graph. This way, you’re building velocity AND complexity simultaneously.

- Monitor the health and progress of a testing program over time.
- Use as a conversation and alignment tool for a team, so that they have a way to measure progress.

Is the result you found in a test repeatable?

THIS is the gold standard of scientific research. 3 Methods for Confirming Test Effects Blueprint provides the most common methods of cross-validation. 

Note that holdouts can be difficult to maintain, results also have to be accurate and there can be reliability issues there. It also sacrifices the solution benefit while running. On the other hand, continuous holdouts can lose attribution of any false positives, but they are easier to maintain.

The first alternative is flip tests, where you implement the winner and then rerun the test by removing the winner. Flip tests are probably the easiest to implement and most common to use, especially on a test-by-test basis. But, they got a burning question inside. What if it loses? 

Going backward can sometimes erode trust in your program. For example, when it gets flat, loses, or gets a different type of result. But this is a part of flip tests. If you’ve got a good program and experimentation culture to handle that, you’ll be fine.

The last solution is time series and moving averages. The point of time series and moving averages is that you implement a test and see what happens over time. But you gotta be careful. There are lots of confounding variables here. You can try using the GA effect tool that allows you to do this more academically

- Report on the ROI of a test initiative or group of tests.
- Be extra confident in your test result.

Problem-Statement Focused Hypothesis

Problem-Statement Focused Hypothesis Blueprint helps you ground experiment ideas (or solutions) in research, utilizing 'problem statements' as the bridge. This enables you to ensure your tests focus on problem statements, which are grounded in research and allow for alternate 'solutions' to be proposed as long as they are both grounded in the same hypothesis (and problem statement).

Let’s say you have a concrete, tactical test idea. With this framework, you can put this idea into the solution part, and then find your hypo and if statement (from that idea). What do you believe will happen if you implement your idea? Now, it’s time to take a step back. What is your problem statement? Where is the evidence that your test idea is really a problem? Most of the time, you can back this in research.

You can also link these problem statements to the business. Use them as an opportunity to understand what your business is trying to prioritize. This way, when you present in front of the leadership, you all can collectively agree on which three problems should be addressed first, instead of having a bunch of solutions backed by hypotheses.

- Prioritize your tests based on business needs.

- Connect your solutions to business problems.

- Get buy-in for experimentation.

- Focus on the most important user problems.

Research XL

RXL Blueprint is a research method for identifying the key barriers to conversion and key customer problems within UX. It is a really strong foundation for any experimentation program. Whether you’ve been testing for years or you’ve never done a test before, or for anyone in between, RXL provides us with a deep understanding of what really matters to your customers and onsite users. This way, you can design tests with impact.

Perhaps you’re struggling with testing lots of random things or your stakeholders are asking you to test a lot of random things. ResearchXL helps you move away from this random approach and base your testing decisions on user data. Now, you have an alternative to stakeholders asking about key customer problems or doing random tests. Ultimately, with RXL you will understand your customers a lot better, with clear benefits for your company.

- Plan UX research.
- Structure your testing and back it in research.
- Identify and classify the fears, frustrations, and motivations your users experience.

Multi Armed Vs A/B Testing

Multi-Armed VS A/B Testing Blueprint is a guiding tool on when to run a multi-armed or a true A/B test. A/B testing allows for a more statistically controlled learning environment, while MAB is more focused on generating a win as quickly as possible (at the sacrifice of understanding 'why'). MABs are good for holiday, short-term, and seasonality testing, while the A/B test provides a deeper insight into what went good or bad in your tests.

- Decide if and when to use MAB or AB.

A/B Testing Workflow Map

This is an example of a workflow map for an A/B test. The different steps right before a test goes live, during the test, and afterward. Each step can be customized based on your organization's structure and needs and can be more granular or less granular.

• Create a workflow map that lays out the rules and steps for setting up a test, the steps while the test is live and stopping decisions, post-test analysis flows.

• This document would be used across the teams that run A/B tests across your company, or if you are an agency you want to map one out for each client individually.

• This would list out steps, tasks, and decisions. For example, if something is not working, what is the fallback.

Sample Ratio Mismatch (SRM)

Sample Ratio Mismatch Alert! What do I do?You have set the ratio for your test at 50/50 and your testing tool has reported that you have a 70/30 traffic split in your experiment. This can not be trusted and needs to be rectified if this experiment's data is to be analyzed. There are several ways that your experiment can have a sample size mismatch, but the most common will be technical issues with the segmentation itself. Use this framework as a guideline to determine what is the best next step once you have received a sample ratio mismatch error:

The CRO Process

The CRO Process blueprint is your strategic approach for identifying and interpreting relevant data to find possible points of friction in your sales funnel. And ultimately, increase the conversion rate. This blueprint shows you all the parts your CRO process should have, including which role should be responsible for which part of the process.

- Structure your CRO Process.
- Improve conversion rate.
- Help everyone understand their place.
- Increase website ROI.

At what point do you move on and start testing something new?

When does it make sense to continue iterating because there’s still some juice left to squeeze? The Iterate Vs Move framework deals with this.

This is a great visualization of how iterations are important to experimentation, but can sometimes be deprioritized if other initiatives take precedence. Remember, testing and iterations are costly. Use this blueprint as a starting point in decision making whether to iterate vs. move on to the next test, and build this into your own prioritization framework.

Use Cases:

- Determine before the test if the iteration costs too much.

- Decide between iterating and moving to a new hypo.

‍

Test Phase Gates

Our biggest and boldest blueprint. The Test Phase Gate Blueprint deals with the test’s phases, stages, and process itself. It helps you ask vital questions to yourself and your team as your experiment goes through different stages. The Test Phase Gate Blueprint is at the heart of experimentation program management.

This is a BEAST of a blueprint, not one to look at every week, but one to help check and balance how your Gates are working (or not), what questions, activities, deliverables, and use cases, are for each gate, and more. It puts a LOT of things into context. The blueprint also references the artifact in terms of building a test document and having all of those pieces sorts of stack up with each other.

The gates represent experimentation programs or parts of their flywheels which, once you turn them on, become one-way gates with no turning back. Check out the Miro board below in ‘related links’ for more info.

Use Cases:

- Manage your experimentation program.
- Determine the cadence and flow of your experimentation flywheel.
- Use as a communication tool to align the team on how things work.
- Understand the artifacts (test documentation) and how they fit in with each other.
- Understand the roles, activities, deliverables, and responsibilities in the flywheel.

‍

‍

So you have a winning test. Should you implement it?

Don’t jump straight to yes. Not all changes are good. Especially if they involve changing the process for several teams or serious stakeholder buy-in. Test Implementation Checklist Blueprint gives you a plan for attack when you have to change ‘business as usual’. 

More than anything, this is your ‘flicker’ mechanism. Something you check on every once in a while and ask yourself a series of questions about the technical feasibility and buy-in you need even before the test is scoped.

Speero's Test Implementation Checklist - Google Slides

Use Cases:

- Have a plan of attack when changing the status quo.
- Guesstimate how much buy-in and technical feasibility you need to implement the test.
- Determine if the test is worth implementing before you present it to the stakeholders.

Experimentational Organization Chart

One of the most challenging questions we hear from companies looking to increase experimentation maturity is how can they better build their teams and distribute their experimentation capabilities within the org. There's obviously not a single answer to this question, but the Org Charts blueprint will present you with the most common examples we see in real life. Org Charts Blueprint is inspired by Stephan Tom's “Experimentation Works" book, which is a great reference on this topic.

Use Cases:

- Structure the experimentation capacity and capability within the organization.
- Structure experimentation teams.
- Establish the responsibilities of the team members for increased efficiency.
- Understand the pros and cons of each org structure.

RASCI

When you start assembling an experimentation team, either by hiring new people or just asking people already part of the company to support your experimentation activities, it is easy to struggle to define who's expected to do what in a specific activity. Who's responsible for this and who's accountable for that? So the idea behind introducing an Experimentation Program RASCI Matrix Blueprint, which is a known tool in the program management world, is to visually represent:

— Who's responsible?

— Who's accountable?

— Who supports?

— Who's consults?

— Who's informed about each one of these activities?

Use Cases:

— Structure experimentation teams.

— Establish the responsibilities of the team members for increased efficiency.

— Structure the experimentation capacity and capability within the organization.

‍

‍