Welcome to Sideshow episode 4, a new Speero podcast where we interview operators and leaders of growth, research, and experimentation every week about:
- What’s working these days, the latest techs and techniques.
- Where’s our industry going? How’s AI changing it?
- Experiments tested or insights learned
This time, we swapped questions. What if instead of asking, "How mature is our experimentation program?," we asked, "How does the structure of our organization influence how well we experiment?"
This question is at the heart of a new paper I co-authored with Nils Stotz, guest on this episode, and Lucas Vermeer, a huge credit to both of them for pushing me to get this done.
We've been geeking out on this for a while, and the paper lays out a framework to help you understand where your company fits in the grand scheme of things. It's a way to put yourself on a map and figure out how to get to where you want to be.
The research for this came from a ton of experimentation org structure interviews I did and a massive Miro board I put together. The interviews were with people from known experimentation companies like Spotify, Vista, Hulu, AMEX, Booking.com, and Disney.
Take a look at all the 10 interviews in the Miro board, along with AI summaries of each interview and accompanying resources.
TLDR:
- The maturity of your program isn't just about the tools you use; it's about how your organization is structured and how teams work together.
- A new framework maps companies into four quadrants based on their organizational structure (centralized vs. decentralized) and their operating model (product-led vs. feature-led).
- A CoE is a powerful tool to transition a company from a "feature-led" mindset to a more ideal "product-led" model. Its role is to standardize processes and act as an evangelist for experimentation across the organization.
- Change management is crucial. A company's culture—whether it's more focused on process or on application—influences how it adopts and scales experimentation.
- AI is changing the "how" of experimentation, offering new ways to automate tasks, surface insights, and orchestrate the entire process. This will further enable experimentation to become a fundamental operating system for business growth.
How You Work and Organize Your Teams?
Once I started putting all organizational structures on a graph, the patterns became clear.
The x-axis is your organizational structure (how you organize your team), and the y-axis is your operating model (how you work). The two axes create four quadrants, and we've defined what each of these means for an experimentation program.
This framework is a big contribution because it gives us a shared language. It lets us stop asking a vague question about maturity and start asking a more direct one about structure and process. Every company, from the smallest startup to the biggest enterprise, can be placed on this map.
Here’s how the quadrants break down:
1. Centralized & Product-Led (Top-Left): The Default Startup.
Most companies start here. A small team, a founder or CEO close to the customer, and everyone doing a little bit of everything. This 'startup nature' makes them centralized and agile.
There's no need to decentralize because the org isn't big enough yet. The product operating model is optimal, and experimentation is a core part of how they work. There’s no siloed work or functional excellence.
2. Decentralized & Product-Led (Top-Right): The Hyper-Growth Ideal.
This is the "optimal" situation described in Lucas's flywheel paper. Companies like Booking and Uber come to mind. Here, you're both product-led and decentralized, meaning every team has the opportunity to experiment on their own without needing much support from anyone else.
Experimentation is a continuous part of the product development lifecycle; it's just how they ship, build, and decide what to build. They don't have boundaries and have power to run bigger bets as strategic decisions.
The McKinsey research we looked at found a correlation between this way of working and business results and innovation. It's a strategic decision to operate this way.
3. Centralized & Feature-Led (Bottom-Left): The Waterfall Trap.
This is where a lot of companies fall as they grow. They move from being product-led to feature-led. Instead of using experimentation to explore and discover, they use it to validate something they've already decided to build.
This can lead to wasted resources and a lot of inefficiency. Imagine the "go build this" mentality, where teams are more like cogs in a machine.
4. Decentralized & Feature-Led (Bottom-Right): The Wild West
This quadrant is an interesting one. Here, the organization is decentralized, meaning individual teams have autonomy, but they are still feature-led in their approach.
This can create a situation where a bunch of individual experiments are going on, but there’s no unifying operating model to connect them. It can be a place of disconnected efforts and siloed decision-making where the company as a whole is not set up to learn from its experiments.
This can sometimes be seen in a general management or business unit model, where every unit is built on its own with its own stack and workflows, similar to Amazon's early days.
The Center of Excellence: A Bridge to the Top-Right
Making the leap from the bottom to the top-right quadrant is really hard. It’s not something that just happens. This is where a Center of Excellence (CoE) comes in.
A CoE is a great mechanism for transitioning your org. It's a central hub that helps you standardize language, tooling, and processes across the company.
It's a way for leadership to say, "Experimentation is important, and we're going to get it into all the different teams".
A CoE needs a cheerleader to advocate for it and an operator to enable it. The goal is to onboard teams and make them aware of what the CoE can do for them.
However, this can go wrong. A CoE can be seen as an elitist group that "tells you how to experiment". This can lead to resistance and a lot of debate, but it's often the best first step for a leader who wants to transition to a more experimental mindset.
I like to think of experimentation as a Trojan horse. You introduce it, and it can become a virus that changes how a business operates.
The body might try to reject the virus at first. A CoE is a way to say, "No, this is good for us. This is our gut flora. Let's learn to live with this and go forward".
The Cultural Divide: Process vs. Application
This also touches on a really interesting cultural divide. As an American, I believe we are a child of Francis Bacon, focused on inductive, application-based thinking. Europeans, on the other hand, are children of Descartes, rooted in deductive, process-heavy reasoning.
I believe this means the term "Center of Excellence" has a different meaning depending on the culture. A CoE might work better in Europe, where the emphasis on process is already there. In the U.S., you might not even use that language.
You would just say, "We need to standardize our language and our tooling," and then you just get it done. It's more about the outcome and the application.
But no matter where you are, change is hard. You can't just introduce a new tool or a new mindset and expect people to embrace it. It takes a lot of time and effort.
Change management is a skill we all need to focus on, and this framework helps you identify the different shifts you'll need to make to get your organization to where it needs to be.
What’s Next?
This paper is just the beginning. The next step is to quantify what we've seen and get more data to validate these different organizational paths.
I'm also doing a new interview series on how companies implement complex experimentation tools and how they make sure the entire organization leverages them successfully.
We've talked to teams at Zalando, Vista, and Marriott to see how they’ve handled these implementations. We're looking at what happens after you "buy a Cadillac" and how you expand that knowledge and integrate the tool into your organization.
Ultimately, this all comes back to a core idea: experimentation is a superior operating system for growing a business. It's not just a tool; it's a way of working. And our goal is to help more organizations see that and make it a reality.
.svg)
