Before we start. This is part 4 of the Center of Excellence interview series between Ben Labay, Speero’s CEO, and different experimentation leaders from small and big companies like RS Components, Disney, Hulu, Vista, Booking, and AMEX. You can find all the interviews and accompanying resources in this free, ungated, Miro board.
All to explore the intricacies of structuring and scaling experimentation within top tech companies. This time, he sat down with Rommil Santiago, a seasoned expert who began his experimentation career 16 years ago, holding over a dozen testing and analytics roles.
Or read any blog post/interview of the series:
- How to build the right CoE with the Right Tech, Process, and Contract
- Three Key Steps to Launching CoE
- CoE: Change Starts Everywhere
- Why One Structure Doesn't Fit All Companies
- How CoE can work like a Charm in a Brand with 6K Employees
It feels like everyone's talking about experimentation team structures. Pods, squads, Centers of Excellence (CoEs) — it's a lot to keep straight, and figuring out what works for *your* company can feel like finding a unicorn.
After managing 10 experimentation programs from small brands to big giants, Rommil is here to tell you why you shouldn’t chase the unicorn anyway.
It Really Does Depend (and That's Okay)
Rommil doesn't beat around the bush: "The structure ultimately depends on the company. It looks like a cop-out, but it's the honest truth.”
He’s seen it all: pods, CoEs, and even places where experimentation is just part of everyone's job. “The big thing here is culture. Are product managers really accountable for their experiments, or are they just checking a box because someone told them to?” That's a huge difference.
The Center of Excellence: A Double-Edged Sword
Once upon a time (recently), Rommil worked at one of Canada’s largest retailers. They set up a CoE. The idea was that teams would come to them with questions, and the CoE would coach them, set up tools, guide them on reporting, and build learning roadmaps.
For some teams, the CoE worked great. Product and growth folks would understand experimentation and just needed a few pointers, like, "Okay, I got it, this is how we do it." Easy.
But others wanted more handholding. And here's the thing: as people moved around within the company, the CoE found themselves teaching the same stuff over and over. Soon, they had too many people to support for a small CoE.
This is where it gets tricky. Some business lines needed really detailed help. We’re talking about things like customer research, figuring out how to analyze a funnel, and understanding past business context. This turned into someone saying "The CoE is nice, but we really need you embedded with us, as a part of our product team."
So, they tried it. They sent a couple of CoE folks to work directly with product pods, attending sprints and ideation sessions. This was helpful for those specific pods, but it meant those CoE members couldn't support the rest of the organization.
Here's what Rommil found: there’s a tipping point where a CoE can get too spread out and can't get into the weeds. But if you spread everyone out in a federated model, knowledge sharing becomes harder. And that's why, it really does depend. That’s ok.
This is where the Org Charts Blueprint comes into play. It highlights the need for clear roles, responsibilities, and resource allocation to prevent these backlogs and ensure the CoE can effectively support diverse teams.
Product vs. Project Operating Model
Ben Labay then brought an interesting point: there’s a pattern emerging in how organizations work. You have product operating models, like Disney or Spotify, where autonomous pods own specific touch points and have all the design, development, and other functions they need built right in.
Then there's the project-based operating model, where you see functional silos. Think marketing-led, sales-led, or IT organizations where requests get tossed over to a development queue. The success or failure of a CoE often comes down to this underlying organizational structure.
The reality is, getting enough staffing for a CoE is tough, especially in large organizations. Plus, different business lines might not even be on the same tech stack. So, a CoE team needs people who can jump into apps, web, data services, and whatever else comes their way. Finding those "unicorns" is super hard.
This is where Experimentation Program Management comes into play. It highlights the need for clear roles, responsibilities, and resource allocation to prevent these backlogs and ensure the CoE can effectively support diverse teams.
Rommil agrees that “a CoE can work, but only if they have dedicated resources.” Otherwise, you end up with a huge backlog of work, especially if the CoE has to juggle requests from many different stakeholders and business lines. Then you have to figure out which work is more important, then play politics to prioritize it, while all those people… don’t like waiting.
Incentives and Accountability: The Real Drivers
Ben Labay noted that a CoE should be working itself out of a job by training and supporting product owners to run their own experiments. This involves:
- SOPs,
- Documentation
- Training materials
- And standardizing tech stacks.
But there's another crucial piece: incentives and accountability.
Rommil has seen it firsthand: some areas received CoE help well, others not so much. The big question is about accountability systems, like ROI or metric accountability for these product pods. Should they be incentivized to run a certain number of experiments (output), or should they be measured on outcomes?
Rommil is clear: “you'll see the most success when the teams being helped by the CoE are incentivized by outcomes.” If you're on the hook for retention or revenue, you're going to test like crazy. But if you're only incentivized for delivery, well, then your job is done once it's delivered, regardless of the impact.
Here's where issues start: what if you have a mix? Some groups are incentivized for revenue, others for feature delivery. How does the CoE, sitting in the middle, weigh those against each other? You end up with a "mishmash of KPIs" for the CoE. If your organization isn't all aligned, you'll constantly have prioritization problems, or someone will have to make tough calls on what's more important.
This is precisely why the Experimentation Program RASCI Matrix blueprint emphasizes defining who is Responsible, Accountable, Supports, Consults, and is Informed about each activity, helping to establish responsibilities for increased efficiency and structured teams.
The Power of Rituals: Retrospectives as a Forcing Function
Rommil doesn't give specific artifacts, but highlights a powerful ritual: the retrospective. If you're familiar with product management, you know about regular retrospectives where you talk about what's working, what to continue, and what to stop.
But here's why it's especially useful for a CoE, particularly if you have resources embedded in different pods or if people don't talk much: it's a "forcing function" for creating ways of working. You can share what's working well in one place and bring it to another. And you can identify what's *not* working and stop doing it elsewhere.
Forcing this knowledge sharing, even if it sounds simple, teaches you a lot, especially when done every week. You create action items, like developing standard operating procedures and documenting them. This discipline of constantly improving your craft is what helps. It works in the product world, and it works for experimentation too.
While weekly retrospectives might seem intense, Rommil says they don't have to be long—half an hour to an hour, tops. But there’s always something to talk about, always something to learn.
The Cadence for Experimentation Meetings blueprint ties directly into this, providing guidance on how, when, and about which topics you should have touchpoints with your team to align, coordinate, and communicate.
Don’t Just Build It, Understand It
Building an effective experimentation team structure isn't about finding a magical solution (unicorns). It's about understanding your company's culture, its operating model, and how incentives are aligned.
A CoE can be incredibly valuable, but it needs the right resources, clear accountability, and a commitment to continuous learning through rituals like regular retrospectives.
There’s no perfect answer, but by asking the right questions and being honest about your organization’s realities, you can build a structure that actually works.
.svg)
