Experimentation / CRO

Experimentation and Testing Programs acknowledge that the future is uncertain. These programs focus on getting better data to product and marketing teams to make better decisions.

Research & Strategy

We believe that research is an integral part of experimentation. Our research projects aim to identify optimization opportunities by uncovering what really matters to your website users and customers.

Data and Analytics

90% of the analytics setups we’ve seen are critically flawed. Our data analytics audit services give you the confidence to make better decisions with data you can trust.

How to Build CoE with the Right Tech, Process, and Contract

How to Build CoE with the Right Tech, Process, and Contract

Table of contents

Stewart Ehoff is the head of experimentation at RS Group, an FTSE 100 distributor, and service and solutions provider of electrical components, widgets, thingamabobs, and whatchamacallits. 

He transformed a centralized experimentation team into a Center of Excellence that helped 12+ teams start up experimentation, all while increasing velocity and connecting tests to broader business strategy. The centralized team tested 5% of shipped stuff. Now, they’re well on their way to making experimentation modus operandi for the whole company.

This is a story about their journey.

Why You Need a Center of Excellence

A landscape picture of Experimentation Organization Structures Blueprint from Speero
Experimentation Organization Structures Blueprint from Speero

Our CoE journey started three years ago. Back then, we had a centralized experimentation team running tests end to end. The whole team included developers, designers, and analysts. They were doing everything related to testing.

Here’s the kicker: We quickly realized that even though we had high test velocity, our team was covering only 5% of all the stuff the business was shipping. 

The tests weren’t always connected to the broader product strategy, product goals, and roadmaps. We tested 5% of stuff, while 95% slipped through our fingers.

Helpful blueprint: Org Charts — understand 3 main ways to structure your experimentation teams so you can increase maturity and distribute capabilities.

Technology + Processes = Center of Excellence

A landscape presentation of XOS, Experimentation Operating System, Tool Stack Blueprint from Speero
Experimentation Operating System (XOS) Tool Stack from Speero

We wanted to make testing the default operating method of how our business ships stuff. Not just for 5-10%. You can only do this if:

  1. You have a process that lets teams leverage experimentation easily as part of their day-to-day work.
  2. The underlying technology that enables you to embed experimentation within product and engineering teams.

So, our journey toward CoE is a story of those two aspects — process and technology. For instance, moving from an internally managed client-side tool to a server-side tool was a big step toward maturity.

We also created all the governance processes — how product teams can use experimentation to reach their goals, reach them faster, and with more evidence. Kill stuff that doesn’t work, and more.

We even created a step-by-step, paint-by-numbers process for taking teams from zero to one with experimentation. It’s basically an agreement between us and product teams, speaking about:

  1. Who’s responsible, accountable, and consultable for what?
  2. What’s experimentation and what value does it bring?
  3. Product teams' KPIs, priorities, and strategies to reach them.

Helpful Blueprint: XOS Tool Stack — glance at all the specific tools you need to start and run effective experimentation programs.

Scaling Experimentation, One Squad at a Time

A list of experimentation program managment metrics
Program Managment Metrics Blueprint from Speero

It's very difficult to go from a fully centralized team where you do everything, to giving away the keys to the city to everyone else and say: now everything is up to you. That's a really harsh transition. You can't cut through it in a week. 

So we started slow, with one squad like a demo version, while the central team continued working as before. This allowed us to learn and keep testing while we scaled experimentation across the company. 

Now we work with 12 different squads across the company and there’s no centralized model anymore. The Center of Excellence is a globally recognized business function across the organization. We work within the EMEA, APAC, and Americas regions spanning 31 countries globally.

My team does not come up with test ideas, we don't build tests, we don't ship stuff. All of the experiment velocity within our business is driven through the product teams who own profit, loss, metrics, and goals.

What we do govern is the governance itself. Everyone must use the same tools, technologies, and processes as us. All experiments are recorded in a central operating system and database where you can see everything.

Helpful Blueprint: Program Metrics — monitor the success of your experimentation program, and spot bottlenecks and issues in the fastest way possible.

Center of Excellence Is Not a Support Tool

Besides governing, we also soft support product teams that experiment. Some squads get more intimate support, depending on the situation at hand. Here’s the deal: it’s not our goal to be a first-line support for these squads.

My team takes squads from zero to one experiment. That’s their goal. Only then helping those teams scale and to loosen the reins as they get better at experimenting. 

The idea is that over time, we build full-on experimentation squads who just touch base with us around particular challenges or specific experiments.

But we’re not over our biggest challenge yet—most of the test analysis is done by my team. Currently, we don't have the structure or resources for dedicated analysts in product squads. 

We still can’t delegate the full experimentation process to a squad because we still depend on CoE for test analysis. But we are where we are. 

Currently, we’re looking to scale through this. How can we manage the near-term capacity of a business with a 4x velocity curve? 

We’re also thinking about the future model of our CoE and broader organization. How to construct our squads and align them around metrics, customer problems, and transient horizontal journeys?

Rounded Experimentation Experts Are (Rare) Unicorns

I like to split the experimentation specialism role into two distinct roles:

  1. Experimentation specialists with good communication and stakeholder management skills, some product experience, and lots of experimentation experience.
  2. Data specialists who look after data quality, technology, test validation, test analysis, and guardrail metrics. They meta-analyze the whole program.

This is a hot, unconventional take. But while scaling I found out that finding people who can do both in a single role is hard. Very hard. I know they exist because I know many of them. But they're unicorns. 

This is something we do differently at RS Group. There's no one-size-fits-all solution for how you build your team and Center of Excellence. The conditions of the market and hiring often determine how you develop.

Contract as a Framework

A landscape presentation of Whats In a Strategic Testing Roadmap Blueprint from Speero
What's In a Strategic Testing Roadmap Blueprint from Speero

I'm a big believer in trying stuff out and breaking things, as fast as possible. Then you catch up and learn. We honed this and did it 12+ times by now, so we’ve built something akin to a blueprint/framework.

It's a step-by-step, paint-by-numbers onboarding process. It may not work for other businesses, but for our company, it’s a great education program. On a high level, it answers questions like:

  • What’s experimentation?
  • Why is it important for our company?
  • What value does it bring?
  • How do we operate as a CoE?
  • What are our priorities and KPIs
  • How to reach those priorities and KPIs?

Sometimes when you engage with other teams (especially at the beginning), people get a bit defensive. We often show them that we’re to help them achieve their goals faster through experimenting, and that quickly goes away.

But if you don’t help them in the right way, it may seem like you’re trying to take away their stuff/product. This way, they don’t feel like we’re here to kill their darlings.

If we were to engage our squads without this, everything would fall apart because there’s no agreement. We wouldn’t be clear on the goals and KPIs. We wouldn’t be clear on responsibilities and accountabilities:

  • Who does what in the ecosystem?
  • Who’s responsible for specific parts of the experimentation process?
  • Who should be consulted when things go wrong?

When you pull all these different people together across a squad, everyone knows where they stand at different parts of the process. Where they add value and where they can (perhaps) step away.

Helpful Blueprint: What’s in a Strategic Testing Roadmap? — a great OKR-style action plan for a testing program and a punch list of insights that help you move the needle on goal-associated KPIs.

Did you know we organize workshops for increasing the maturity, speed, and impact of your testing/experimentation programs? Feel free to take a look.

Related Posts

Who's currently reading The Experimental Revolution?