One thing before you read. This is part 7 of the Center of Excellence interview series between Speero’s CEO, Ben Labay, and a dozen experimentation program leaders from cutting-edge companies like Hulu, Vista, Spotify, AMEX, Disney, Booking.com, and more.
All in the effort to collect relevant thinking, resources, and experiences on how experimentation and team organization and structure intertwine, and how this links to organizational operating models like the product operating or project management models.
Feel free to explore the ungated, free Miro board that has all the interviews and their resources, or read/listen to any interview from the series:
- How to build the right CoE with the Right Tech, Process, and Contract, with Stewart Ehoff, head of growth platforms and product operations at RS Group.
- Three Key Steps to Launching CoE, with Melanie Kyrklund, global head of experimentation at Specsavers.
- CoE: Change Starts Everywhere, with Ruben de Boer, lead experimentation consultant at Online Dialogue.
- Why One Structure Doesn't Fit All Companies, with Rommil Santiago, founder of Experiment Nation and Sr. Director of Product Experimentation at Constant Contact.
- How CoE can work like a Charm in a Brand with 6K Employees, with Luis Trindade, Principal Product Manager of Experimentation at Farfetch.
- Hitting the JIRA Wall–Why Waterfall Fails Experimentation and What Works Instead, with Dan Layfield, director of product management at Diligent, ex-Uber.
- Centralized Teams Can Work Wonders in Small Businesses, with Kevin Anderson, Sr. Product Manager of Experimentation at Vista and the writer of the Experimental Mind newsletter.
This time, Ben interviewed Kevin Anderson, Sr. Product Manager of Experimentation at Vista and the writer of the Experimental Mind newsletter.
The question of the day is: Do you centralize all experimentation efforts, or do you embed them directly within product teams? And how does that choice impact how your organization operates?
Kevin experienced both ends of the spectrum in his career – from a highly centralized model at ING to Vista's vision of fully empowered product teams. His answer? Centralized teams can work wonders in small businesses, but for bigger companies…
The Fully Centralized Model: Early Growth and Efficiency
At the start of his career at ING, Kevin saw a "fully centralized model" in action. A single person, then a small team of two, handled A/B testing end-to-end, from hypothesis generation to experiment setup and analysis.
This dedicated approach proved powerful, enabling them to run "over a couple of hundred tests per year". These setups can be incredibly efficient in the early stages or for smaller organizations: “If you have good, dedicated people and the right tools, you can move very fast and very far".
Many organizations today still operate this way, and for good reason. It allows for deep specialization and consistent application of best practices from a single source.
The Empowered Product Team Model: Scaling for Quality
These days at Vista, Kevin is at the other end of the spectrum. “The central team doesn’t do any experimentation itself, but only supports, empowers, and enables the rest of the org.” This means identifying friction points, solving them, and ensuring that others can run high-quality experiments at scale.
This shift to the “empowered product teams model”, often operating under a "product operating model," means that product teams are set up to own their problem space, define their metrics, and measure their own success.
Such a shift also tackles challenges like resource allocation and setting up teams for their own accountability because teams are staffed and aligned to specific problems and their tracking.
For the central experimentation team, this makes it "far easier to connect those two things together," focusing on more implementation-based questions like tool setup and aligning tracking with team goals.
A helpful blueprint here is the Experimentation Decision Matrix blueprint. This blueprint provides an action plan before starting any testing, based on primary and secondary success metrics.
It helps classify results based on hypothesis type and the action to take afterward, enabling a narrative where "winning" and "losing" are not necessarily what's impactful for the business (it's actually aligning actions with desired changes….e.g., are we solving the problem we wanted to solve?).
The Center of Excellence: A Bridge to Empowerment
Kevin's experience at Vista, where the empowered product team model was run in parallel to the development of the Center of Excellence, highlights a vital point: the CoE often acts as a bridge. It helps an organization transition from a top-down, project-managed approach to a more bottom-up, empowered product culture.
Here, Ben asked an interesting question: “What’s the evolution of CoE? What happens to it when it's successful? Does it work itself out of a job?” The CoE embeds experimentation into empowered teams, who now need the CoE less and less as they experiment on their own.
Sidhant Bakhru from Hulu/Disney talked about how they disbanded their CoE because product teams experimented completely on their own. There must be a version of the story where the CoE team becomes a management/leadership team helping with strategic direction, people management, recruiting, and more.
Kevin agrees that it "might be the next step in the evolution" for Vista. Currently, Vista’s CoE empowers product leaders with "the right set of evidence, the right set of data" for product iteration, with insights bubbling up from the product teams themselves.
As experimentation becomes more deeply embedded there, the CoE might shift from enabling quality experiments to helping product leaders with strategic direction, management, and resourcing.
But for Kevin, in the end, “the insights and learnings should come from the product teams, and bubble upwards toward the product leaders, which informs the future offers”.
While talking about strategy and direction, one of the most helpful blueprints for both is the Strategic Testing Roadmap blueprint. The blueprint is a "culmination of research and the basis of a great OKR-style action plan for a testing program", focused on creating a research-based strategic roadmap and communicating objectives and key results.
Metrics and Accountability: The Linchpin of Empowerment
For empowered product teams to truly thrive, clear metrics and accountability systems are essential. Ben thinks that “experimentation is a Trojan horse for creating empowered product teams”, largely because it forces clarity around metrics and accountability. It’s about letting teams autonomously experiment, but also keeping them aligned with the organizational strategic goals.
For Kevin Anderson, the “starting point isn't the metric itself, but rather customer problems". This lets you decide which domains and product teams you should create.
Once the problem space is defined, product teams have the freedom to figure out "what are we going to measure and how does it ladder up to the business KPIs on the highest level".
The process of mapping top-level KPIs to individual team metrics can be extensive. At Vista, it took their product analysts about six months to create this clear mapping across the entire organization, from individual domain and product team metrics to business goals.
But this provided insights “for the central team to map specific experiment and product teams KPIs”.
This comprehensive "metric system" becomes the "foundation of the things that you want to optimize for and the things you want to measure". Each quarter, teams then define their OKRs (Objectives and Key Results) based on which metrics they want to push.
This approach directly ties into Speero's Goal Tree Mapping blueprint, which provides a visual framework for defining strategic KPIs and breaking them down into smaller, actionable metrics. It lets you align teams to a common goal and ensures everyone understands how their efforts contribute to the company's success.
The Experimentation Meeting Cadence blueprint also supports this, outlining how regular meetings (weekly, monthly, quarterly) can be used to create feedback systems, impact culture, and ensure accountability within experimentation programs (Kevin: “We also have a set of KPIs we look at every month, but also each quarter we pick one area to focus on”).
Fully Centralized or CoE?
After running several interviews with CoE leaders, Ben Labay created a chart of experimentation vs org structure, later updated with Product Operating Model vs. Feature/Project Management Ops model.
Here’s what Kevin said about team structure and operating models. He characterized his early experience at ING as being in the "fully centralized product management corner". While ING has since moved towards "empowered agile teams and product operating model," it's not always a linear progression; organizations can "bounce back a bit".
For Vista, Kevin places them firmly as a "Center of Excellence" on the team structure axis, noting that while there are some decentralized pockets like pricing experimentation, the core is still CoE. On the operating model axis, they are "definitely the product operating model," having fully embraced the empowered product team approach over the last year.
Ultimately, there's no single "best" model. It all "depends on the context and what your goals are".
.svg)
