Welcome to the last part of the interview series between Speero’s CEO, Ben Labay, and vital experimentation leaders from key global organizations like Disney, Spotify, Hulu, Booking(com) and AMEX.
His goal? Research how these orgs scale and structure their experimentation efforts, especially when it comes to the Center of Excellence (CoE) model.
Find all the interviews and resources on this Miro board. Other interviews on the list include:
- How to build the right CoE with the Right Tech, Process, and Contract, with Stewart Ehoff, head of growth platforms and product operations at RS Group.
- Three Key Steps to Launching CoE, with Melanie Kyrklund, global head of experimentation at Specsavers.
- CoE: Change Starts Everywhere, with Ruben de Boer, lead experimentation consultant at Online Dialogue.
- Why One Structure Doesn't Fit All Companies, with Rommil Santiago, founder of Experiment Nation and Sr. Director of Product Experimentation at Constant Contact.
- How CoE can work like a Charm in a Brand with 6K Employees, with Luis Trindade, Principal Product Manager of Experimentation at Farfetch.
- Hitting the JIRA Wall–Why Waterfall Fails Experimentation and What Works Instead, with Dan Layfield, director of product management at Diligent, ex-Uber.
- Centralized Teams Can Work Wonders in Small Businesses, with Kevin Anderson, Sr. Product Manager of Experimentation at Vista and the writer of the Experimental Mind newsletter.
- Backtest, Incentivise, and Watch Out for Variance, with Liam Furnam, data scientist at RealFi (ex-Booking, ex-Meta).
- Replacing CoE with an Experimentation Guild, with Tim Thijsse, CXO specialist at Online Plastics Group, Ex-Beerwulf (Heineken subsidiary).
- Get Your Head Out of the Sand, with Marty Cagan, writer of Inspired, Empowered, and Transformed, and partner at Silicon Valley Product Group.
This time, Ben sat down with Marty Cagan. If you're in the product or tech world, his work on product management has been gospel for years. But for those of us in the experimentation community, his name isn't always top of mind. And that's a problem.
Ben and Marty talked about everything from why you can’t optimize your way to growth, why innovation beats predictability, how CoE is the happy medium, and much more.
His message to our world is pretty simple and direct. Get your head out of the sand.
TLDR:
- True experimentation goes beyond simple A/B testing and conversion rate optimization (CRO). Marty Cagan argues that a sole focus on low-risk optimization is a "bug, not a feature" of a company's system, and it prevents real innovation.
- The biggest dimension of experimentation is often missed: high-risk, qualitative experiments that lead to significant innovation, like the foundational work for Amazon Prime. These should complement quantitative testing, not be replaced by it.
- Many companies are "addicted to process," which leads to predictable, low-risk work instead of innovation. Cagan advocates for a principles-based product operating model where experimentation is a core component.
- The experimentation community is uniquely positioned to drive organizational change. Their culture of accountability and reliance on data can be used as a "Trojan horse" to shift companies from a project-based, output-focused model to an outcome-based product model.
- A centralized Center of Excellence that runs every test is inefficient, as is a completely decentralized model where every team works alone. Cagan proposes a "happy medium": an expertise-based CoE that acts as a coach, guiding and enabling product teams to do their own good work.
- Agencies that simply run experiments for you are a "complete waste of money." The goal is to institutionalize experimentation skills within every product team so that it becomes an ongoing part of their culture, not a temporary phase.
The Problem with Optimization-Only Experimentation
In the experimentation world, we often think of what we do in terms of A/B tests, funnels, and conversion rates. “And that’s a good thing,” Marty says. But it’s not the whole thing. He calls these small changes “optimization experiments.” And they are a part of a much larger picture of "product discovery."
Marty sees a lot of companies where experimentation = optimization. This happens because either they don't know how to do bigger experiments, or they're just scared to take bigger swings.
When you just optimize, you are not a product manager. Marty believes that all product managers should be skilled at doing the entire job, which includes both optimization and innovation.
He feels that calling someone an "optimizer PM" is just "a rationalization of the crappy work out there". It's a "bug, not a feature" of the system.
This is where the Solution Spectrum Blueprint can help you. This tool allows you to categorize your tests as Iterative, Substantial, or Disruptive. This can help you see if you're stuck in a rut of low-risk, iterative work and push you and your team to think bigger.
The Missing Dimension of Experimentation
Marty argues that most people in our industry are missing the biggest dimension of experimentation. We’re great at a lot of things, but we often focus too much on low-risk experiments—the "blue to green" changes—and not enough on the high-risk ones that lead to real innovation.
He gives the example of Amazon Prime, one of the biggest innovations in history. Did they use A/B tests? Yes, of course. But that's not how they came up with the idea. The real innovation came from bigger, harder experiments in the back-end, such as:
- Looking at different ways to do shipping
- Changing the part of the shipping Amazon controls
- Analyzing the part of shipping the shippers like FedEx do
“That’s how you innovate.” This isn’t just about quantitative tests, either. Marty points out that a lot of times, experiments are qualitative. You can learn a lot from a quick qualitative experiment, but you don't know what you’ve learned until you back it up with a quantitative one.
He talks about the difference between value capture and value creation. Optimization is in the value capture bucket—making what you have work as well as it can.
But the really hard part is value creation, which is making what you do significantly better. You need both.
Marty has seen numerous big companies that “don’t know what it means to experiment.” But they can’t believe they don’t experiment.
So they make their teams run simple optimization work, even if they have experimentation experts at their disposal. On the other hand, those big companies outsource innovation through mergers and acquisitions.
This is a great place to use the Problem-Statement Focused Hypothesis Blueprint. This framework helps you move away from tactical test ideas and ground your work in research-backed problem statements.
This allows you to connect your experiments to bigger business problems and innovate on how to solve them.
You’re Optimizing for Predictability, Not Innovation
Marty believes that a lot of companies are addicted to process. And even though Europe is the usual suspect, he says this isn't a US versus Europe thing, but a fundamental problem that can happen anywhere.
In a product operating model, the entire system is optimized for innovation, and that's why experimentation is at its core. But most companies don't do real experimentation; they just do the small, predictable stuff.
“A process is an alternative to the product model because it's all about optimizing for predictability, not innovation.” Most companies use processes for consistent predictability, not to disrupt or innovate. While processes “aren’t evil”, true innovation comes from principles, not processes.
This was a serious problem even in 1995. Go and see The Lost Interview with Steve Jobs, where Steve talks about the "disease of process people" taking over and how it led to a lack of innovation.
At that time (1995), IBM used to be this great leading company, then they just “lost all their Mojo”. For Steve Jobs, the Mojo was lost because the processes took over. “In the end, process is an addiction for the company” – Marty Cagan.
This is a great time to look at the Org Charts Blueprint. This blueprint lays out the pros and cons of different organizational structures, like centralized, decentralized, and the Center of Excellence model.
It can help you think about how to structure your experimentation teams to enable innovation, not just process.
Experimentation As the Trojan Horse For the Product Model
Ben mentioned to Marty that he often sees a layer of process-addicted people between experimentation leaders and the C-suite, which makes it hard to get buy-in for bigger, innovative changes.
Marty agrees that you can't implement the full product operating model without strong product leadership. Most companies have feature teams, not product teams, with roadmaps handed down from stakeholders.
These roadmaps drive everything, which turns them into blockers. In this model, you can only hope for optimization.
Discovery is all about discovering a solution, but that’s already been handed to teams in the roadmap. So there’s no discovery.
Is there still room to optimize? Yes. But optimization is all you’re going to get. And you’ll never reach real innovation. “Real innovation comes from a different approach entirely.”
Marty argues that the experimentation community has the potential to be a "Trojan horse" for this kind of change. Our community has a culture of accountability and uses data to prove outcomes.
This is a key foundation for the product operating model, which is all about being accountable to outcomes, not just output. But how do you transition from optimization to product operating model?
Marty wrote a book, Transformed, which is specifically for senior leadership to help them understand how to make this kind of change. It’s about transitioning from funding projects to funding product teams and giving teams problems to solve, not just roadmaps to execute on.
If you're an experimentation leader, this is where the How To Engage Community and Create a Culture Around Experimentation? Blueprint becomes your best friend.
This blueprint gives you a plan to increase engagement, get feedback, and train teams on testing principles, which is a key first step in getting buy-in for a bigger shift in how the organization works.
Don't Fall for the Double Diamond Trap
For Marty, the Double Diamond framework can lead you astray, because it originates from "process people" who tried to apply a rigid process to experimentation.
The Double Diamond’s misconception is that product teams should spend roughly the same amount of time discovering problems as they do solving them.
This is dangerous and leads to "anarchy" in large organizations, where hundreds of product teams all try to figure out the biggest problems to solve on their own.
According to Cagan, product leadership is what keeps teams aligned and moving in the same direction, not a flat, egalitarian approach. Instead of a double diamond, imagine a pyramid.
At the top, leaders identify the most important problems, and then the product teams are responsible for discovering solutions to the assigned problems. This provides necessary direction and prevents chaos.
Team Structures and the Product Triad
The core of a product team, or the triad, typically consists of:
- A product manager
- A product designer,
- And an engineer (or multiple engineers).
This structure applies to all product teams, although a designer might not be necessary for teams working on APIs, for instance. Cagan highlights that you only need a product manager if you are doing real product discovery.
If you’re only focused on optimizing, you don’t need a product manager since your changes, by definition, are low-risk and don't involve significant strategic decisions about selling, marketing, or legal issues.
Your broader team topology—how you organize teams—is a more complex discussion, where you have trade-offs between customers, markets, and technology architecture.
Cagan's book, Empowered, details the responsibilities of product leaders in this area, including product vision, strategy, and how to assign problems to teams.
Cagan noted that a common issue in organizations is that supporting roles like user researchers and data analysts are perceived as "second-class citizens". He believes this is largely an economic issue.
While it would be ideal to have a dedicated user researcher for every team, most of us don't have enough work to keep one person busy full-time. So these roles are often shared among multiple product teams.
CoE Is a Happy Medium
Cagan sees a spectrum of experiments. Low-risk changes, which don't have major usability, feasibility, or viability risks, can be implemented with simple A/B tests. High-risk changes, such as moving a feature behind a paywall or playing with pricing, should be treated as discovery and require a product manager's skills.
Marty estimates that over 90% of the experiments he sees are low-risk optimizations. Regarding team structures for experimentation, Cagan identifies two non-optimal extremes.
One is a centralized Center of Excellence that runs every test, which prevents product teams from learning and slows down the process.
The other is "democratized user research," where all product teams do the work themselves, which can be inefficient because they may miss professional guidance.
There’s “a happy medium,” though: an expertise-based Center of Excellence that doesn't do the work, but helps teams do good work. This small, shared group of specialists can act as coaches to guide multiple teams.
This model is more sustainable and defensible, especially during cost-cutting measures, than embedding a dedicated specialist on every team.
The Last Word: Be the Lab, Not the Test
Marty’s final point was a powerful one: “some optimization agencies are a complete waste of money" because they just run the experiments for you. True outside partners institutionalize the skills themselves.
The goal is that every product team experiments all the time. It’s not a phase. It’s something that never stops.
.svg)
