Welcome to part nine of the interview series between Ben Labay, Speero’s CEO, and key experimentation leaders from prominent global organizations, including Spotify, Vista, Hulu, AMEX, Booking.com, and Disney, among others.
The primary objective is to explore how these companies structure and scale their experimentation efforts, particularly in relation to the Center of Excellence (CoE) model.
Get all the interviews and resources on this Miro board. Other interviews on the list include:
- How to build the right CoE with the Right Tech, Process, and Contract, with Stewart Ehoff, head of growth platforms and product operations at RS Group.
- Three Key Steps to Launching CoE, with Melanie Kyrklund, global head of experimentation at Specsavers.
- CoE: Change Starts Everywhere, with Ruben de Boer, lead experimentation consultant at Online Dialogue.
- Why One Structure Doesn't Fit All Companies, with Rommil Santiago, founder of Experiment Nation and Sr. Director of Product Experimentation at Constant Contact.
- How CoE can work like a Charm in a Brand with 6K Employees, with Luis Trindade, Principal Product Manager of Experimentation at Farfetch.
- Hitting the JIRA Wall–Why Waterfall Fails Experimentation and What Works Instead, with Dan Layfield, director of product management at Diligent, ex-Uber.
- Centralized Teams Can Work Wonders in Small Businesses, with Kevin Anderson, Sr. Product Manager of Experimentation at Vista and the writer of the Experimental Mind newsletter.
- Backtest, Incentivise, and Watch Out for Variance, with Liam Furnam, data scientist at RealFi (ex-Booking, ex-Meta).
This time, Ben sat down with Tim Thijsse, senior CXO specialist at Online Plastics Group. He’s here to tell you what happens when the Center of Excellence model starts to creak under its own weight. When the team is stretched thin and things start to get, well, messy.
He worked for a company called Beerwolf, a subsidiary of Heineken, and their journey from a centralized UX team to a more distributed "Guild" is a masterclass in how to evolve your organizational structure when centralized teams stop being effective.
Reports Going Nowhere
Tim joined a UX team at Beerwolf (a Heineken subsidiary selling beer packages and home draft equipment), which was already doing some solid experimentation work. The brand had a nice system for documenting insights and experiments, but the problem was that all the insights and learnings were just living within the team.
The team would send reports to stakeholders, but those reports often got lost, and the insights weren't really being used to communicate results or evolve the website in any meaningful way.
Tim's challenge was to get that information out to the rest of the company. He quickly realized it wasn't just about sending reports; it was about getting more people involved in experimentation itself. “I quickly realized that it’s not about reports or the information, but what happens after.”
He started looking for "motivators" or "power users" within different teams to pull in ideas and hypotheses from outside his own team, creating more of a pull than a push model. But they hit a wall.
Tim's team was effectively a centralized unit of experts, and they were spending most of their time embedded in other product teams. One designer might be 80-90% focused on one team, and a researcher might be 75% on another. This led to a lot of confusion and frustration.
Designers and researchers had to manage multiple Scrum boards—their own UX team board, plus the product team boards—and report to multiple places.
This is a classic problem. You can have a team of experts, but if they're not fully integrated, they just become a bottleneck. That's a good time to pull out the Org Charts Blueprint. This blueprint helps you figure out how to structure your experimentation teams and distribute capabilities.
The Guild is Borne
To solve this problem, they made a big change. They broke up the centralized UX team and moved the roles directly into the product teams. The UX team itself was transformed into a “Guild”.
So what's a guild? It's not a formal, set team. It's a group of people who share the same interest in a specific topic. In this case, the guild was made up of people “interested in experimentation and using insights.”
This move made the product teams truly multidisciplinary. But it also brought up a new challenge. When you disperse your experts, you can risk losing what's called "functional excellence".
How do you keep everyone on the same page with best practices? How do you make sure a good researcher in one team is sharing their knowledge with a less experienced researcher in another?
This is where the Experimentation Program RASCI Matrix Blueprint comes in handy. It's a well-known program management tool that helps you define who is Responsible, Accountable, Supports, Consults, and Informed for each activity.
It’s perfect for making sure that even when your people are in different teams, everyone knows their role and the flow of information is clear.
The Mechanisms of a Guild (OKRs + Weekly Standups)
Tim and his team came up with some clever mechanisms to make the guild work. Even though they weren't a formal team, they still had their own Scrum board and backlog.
Instead of sharing traditional Scrum stories, they used OKRs (Objective and Key Results). These weren’t about producing a lot of work but were focused on improving the quality of their work (QA) and the process itself.
They kept in touch with weekly stand-ups, and they’d get together every three weeks to check on their OKRs. The most important part was that the output of the guild wasn't a better website or a better experiment. The output was creating frameworks, templates, and guidelines to help the product teams start their own experimentation.
The guild’s job was to facilitate the entire ecosystem. They’d set up a form that anyone in the company could use to submit an experiment idea.
Then, they’d help them with the next steps. The actual building of the experiment, however, happened within the product team, not the guild.
This kind ofzsystem is exactly what the Cadence for Experimentation Meetings Blueprint is all about. It helps you create a ritual pairing of content and timing for your meetings, from weekly tactical stand-ups to monthly process audits and quarterly strategy assessments. It’s a great way to make sure everyone is aligned and accountable, even in a distributed model.
How To Get Buy-in for This Transition
The big question, of course, is how you get leadership to buy into this kind of transition. Moving from a team with its own metrics to a guild focused on quality improvement is a huge change, and it requires different incentives and accountability.
Tim said this “can only happen if the company’s vision and strategy are already aligned with a customer-centric focus.” For Beerwolf, the vision was to focus on the "complete beer drinking experience," not just more sales.
Because of that vision, they were able to introduce customer satisfaction as a key performance indicator (KPI) on a higher level, which trickled down to every team. This meant that the entire company became responsible for those metrics, not just one team.
And because the whole company was already focused on the same metrics, it made it much easier to break up the centralized team and distribute the expertise.
This is perfectly summed up by the Goal Tree Mapping Blueprint. This framework helps you create a logical, visual representation of goals and key success factors.
It lets you align your experimentation efforts with top-level business goals and break down those KPIs into smaller, more manageable metrics that the whole company can contribute to. It's the ultimate tool for getting buy-in and making sure your work matters to the people at the top.
Organizational structures aren't set in stone. They need to evolve with the company, and sometimes, the best way to scale is to let go of control and empower others to take ownership.
Empowered > Centralized
True experimentation excellence blossoms not from centralized control, but from empowered, distributed teams unified by shared purpose and clear frameworks.
To optimize your organizational structure for experimentation, leverage:
- The Org Charts Blueprint and the Experimentation Program RASCI Matrix Blueprint to define roles and responsibilities within your own teams.
- The Cadence for Experimentation Meetings Blueprint to establish effective communication rhythms
- The Goal Tree Mapping Blueprint to align your efforts with overarching business objectives and secure crucial buy-in.
.svg)
