How to Scale Experimentation in Different Structures, Organization Designs, and Operating Models

How to Scale Experimentation in Different Structures, Organization Designs, and Operating Models

There are a million reasons why it's good to scale experimentation. I’m not here to talk about that. You either want to get uncomfortable and grow experimentation in your company, or remain in your zone. I’m here to talk about how the way you scale experimentation highly depends on the organizational design and operating models your organization uses. 

Each model has its own inherent challenges and strengths, which you should consider when you want to scale experimentation. 

These models include:

1. The general manager (GM) org model
2. The functional org model
3. The product operating model
4. The project operating model

When you know how each model behaves—with all the pros and cons—you can customize your approach and scale a lot faster, sometimes without even increasing your headcount (that much). Better still, you won't stress yourself with limitations in your current approach. Knowledge is power.
First, it’s good to know that the GM and functional ORG models are the WHERE and HOW things happen. Check out Lenny’s great article on this topic. On the other hand, the product vs project OPERATING models are the HOW. I like this post from Melissa Perri and this article from Marty Cagan that puts org vs operation models into perspective.

You can have a mix of org models and operating models, and hybrids between them. Almost all organizations are not strict to any model, they operate in hybrid setups with some departments and teams working one way and others in other ways. 

It’s all quite confusing, but there are massive benefits in understanding the core models. Let's get to some definitions, with the implications for experimentation specifically.

Understanding Organizational Models and Their Impact on Experimentation

General manager (GM) model

The general manager model structures product and engineering teams into AUTONOMOUS business units. A general manager leads each one and owns business results and related product mandates. In this model, we focus on speed, agility, and accountability at the business unit level.

The general manager model characteristics in experimentation:

Characteristics of the GM Model in Experimentation

Decentralized experimentation: In the GM model, you decentralize experimentation, letting each business unit run its own experiments, focused on optimizing outcomes specific to its domain.

Speed and agility: Because you gave business units autonomy, they can quickly decide and implement experimental changes.

Local maxima: While you can quickly experiment in the GM model, it often falls into a local maxima—where you optimize for individual business unit goals rather than holistic company-wide metrics. (Helpful Speero blueprints: How to balance test velocity vs complexity, Solution spectrum)

The classic GM model organization

Amazon: The company uses the GM model for rapid production of a relatively big number of new products and lines of business (because they need speed and agility for this). We know the success that Amazon has had with their culture of experimentation. What’s the tradeoff then? Customer experience.

I’ve spoken to and consulted Amazon teams that work inside what we might consider a single ‘product’ at Amazon, and even there they have team silos among channels, devices, and even touchpoints (e.g., marketing vs authenticated experiences). They explain large gaps in uniform customer experiences. (see below how this contrasts with the classic ‘Functional’ example of Apple).

A GM model can devolve into feudalism if left unchecked. Amazon is doing it well. But no one but Amazon is Amazon, thus the pure GM model can be tricky. 

Strategies to Scale Experimentation in the GM Model

Align: align business units to avoid redundant and conflicting experiments. For instance, establish common experimentation goals and KPIs (Helpful Speero blueprints: Goal tree maps, What's in a strategic testing roadmap?).

Share knowledge: Create a platform to share insights and learnings across business units. You won’t only share knowledge this way but also showcase experimentation results better. (Helpful Speero blueprints: How should I structure my (automatic) test reporting? and How to structure reporting across tests?)

Manage resources: You’ll need to balance resource allocation between units (the biggest issue in the GM model) to ensure everyone has the right tools and capabilities to run meaningful experiments. So, define the core standards around processes and acceptance criteria for each gate and phase of the flywheel. Assure standards or otherwise, you’ll have complete chaos. (Helpful Speero blueprints: Test Phase Gate, The CRO process)

The functional model

The functional model centralizes disciplines into functional units, where engineering, product, and design leaders co-own product metrics that drive business outcomes. In this model, you focus on specialization and internal alignment.

Characteristics of the Functional Model in Experimentation

• Centralized experimentation: testing is centralized based on functions, while functional leaders drive experiments to optimize their specific domains.
• Specialization: Deep expertise inside functions drives sophisticated, high-impact experiments.
• Silos: the first issue comes from siloing, where functional departments don’t communicate effectively, slowing down cross-functional experimentation.

The classic functional model organization

Apple: Apple uses the functional model to build and optimize a holistic user experience across a relatively small number of complex products running together in a highly integrated ecosystem.

How do you tell the best designers on the planet that they might be wrong so they should test it? Apple has a north star of something like to be the most creative company on the planet. This is their KPI even within experimentation. They run fewer, but way more sophisticated and well-thought-out experiments.

The tradeoff? Scale. Apple has insane thoughtfulness about customer experience, but experimentation velocity and scale suffer. But this is Apple, they now can afford the best talent, and can get around this perhaps. Some organizations need an experimentation culture to attract talent. 

Strategies to Scale Experimentation in the Functional Model

Enable cross-functional collaboration: To break down functional silos and have more holistic experiments encourage collaboration by setting up cross-functional squads that co-own metrics and touchpoints. I love the example of how Hulu did this.

Integrate platforms: integrated experimentation platforms let you share data and collaborate across functions. You can do this by creating uniform standards for data processing and flows. (Helpful Speero blueprints: How should I structure my (automatic) test reporting? and How to structure reporting across tests?)

Get leadership support: to enable cross-functional initiatives you’ll need functional leaders who can champion experimentation and provide needed resources. (Helpful Speero blueprints: Testing revenue model, Where and how should I test to make the most money)

Transitioning from a GM to a functional model in experimentation

There are a lot of stories of companies changing org models.  Check out this great Lenny’s article on this topic. At Speero, we help consult orgs for how they scale experimentation and have done so for companies like Cisco, Wellhub (formally Gympass), Tipalti, Mongodb, Miro, Clickup, and more.

The most interesting in my opinion is Decathlon. 

We’ve worked with them for 3+ years and ran some testing and research program work, but recently we helped Decathlon transition their experimentation program from a GM model (local/country-based) to a Functional model (centralized standards set by CoE). The aim there was to get a hold of their customer experience and brand. (I’ll be speaking about this alongside their CoE leader at Experimentation Unite in September)

The product operating model 

A reminder that the product operating model is the ‘HOW’ not the ‘WHERE AND WHAT. In my opinion, the spirit of the product operating model is what it solves for or at least accounts for the tradeoffs in the above two org models.

The product operating model organizes cross-functional teams around solving particular problems. Check out my interview with Marty Cagan on the product operating model and its implications for Experimentation.

Characteristics of the product operating model in experimentation:

Integrated Experimentation: You integrate experimentation into cross-functional product teams for rapid testing and iteration directly tied to product outcomes.

Customer-Centricity: in this model you’re highly responsive to customer needs and market changes, letting you encourage high-impact experiments focused on customer behaviors.

Downsides or Yellow Flags

Potential Resource Duplication: Since product teams are independent, you can experience resource duplication and challenges with maintaining functional excellence.

Over-emphasis on autonomy. The product operating model necessitates managers and leaders to act as strong coaches and hire/build self-accountable teams. For the model to work, leadership must fully buy into the concept of empowered teams. However, in many organizations, leaders are accustomed to top-down control and may find it difficult to relinquish decision-making power to product teams.

Companies experimenting with the product operating model include Amazon, Apple, Spotify, and Netflix.

Strategies to scale experimentation in the product operating model 

Here I think experimentation can help with the known or often-cited criticisms of the Product Operating Model:

Unify frameworks: using the same frameworks lets your product teams maintain consistency and quality. Speero’s XOS blueprints can help you standardize various experimentation processes. 

Manage resources: do this to avoid duplication and ensure all teams have access to necessary experimentation tools and expertise. Again, define the core standards for processes and acceptance criteria for each gate and phase of your testing flywheel. (Helpful Speero blueprints: Test Phase Gate, The CRO process)

Constantly improve: AKA systematically capture and share insights from experiments. At Speero we use Zapier automation that systematically post-test results in a Slack channel called “growth-insights” so everyone in the company can see, analyze, and comment on test insights.

The project operations model:

The project operations model organizes teams by project, with each project team potentially including members from different functions. Teams disband after the project is completed.

Characteristics of the project operations model in experimentation

• Project-Based experimentation: Specific project needs drive experimentation, with a focus on innovative solutions for project goals.
• Flexibility: This model lets you stay flexible and allocate resources based on the unique demands of each project.
• Lack of Continuity: The temporary nature of project teams can lead to inefficiencies and challenges in maintaining long-term functional excellence.

Companies experimenting within the project operations model:

• Typically they are Construction firms like Bechtel
• Or Large-scale event-planing or consulting companies like Deloitte and Accenture
• But any enterprise with waterfall practices and functional silos often works on a project ops model.

Strategies to scale experimentation in the project operations model

Keep the knowledge: implement systems to capture and retain knowledge from project-based experiments to ensure continuity and long-term learning. At Speero we keep all insights obtained from experiments, their results, and more in a central repository in Airtable. 

This standardized and searchable database can help your project teams quickly obtain learnings and insights from previous projects that might be relevant to their work. This prevents work duplication and helps with efficiencies.

Resource Coordination: You’ll have to coordinate resources carefully across projects to avoid conflicts and ensure each project has the necessary tools and skills. You can do this by defining the acceptance criteria and core standards for processes across gates and phases of your experimentation flywheel. (Helpful Speero blueprints: Test Phase Gate, The CRO process)

Integrate projects into long-term goals: so you can ensure holistic experimentation and cumulative impact. (Helpful Speero blueprints: Goal tree maps, Strategy maps)

Conclusion

To scale experimentation in different models you need to understand what makes them strong and unique. This is where their issues and challenges come from.

The GM model’s decentralized approach promotes speed and agility, but it can devolve into corporate feudalism if you don’t align teams, manage resources, and share knowledge.

The functional model is all about specialized functions, but you need to overcome silos through cross-functional collaborations, integrated platforms, and leadership support.

The product operating model excels in customer-centric experimentation but needs efficient resource management to avoid duplication, unified frameworks to maintain quality and consistency, and a way to constantly improve.

The project operations model offers flexibility but you will need to find a way to keep the knowledge you get from disparate projects, coordinate resources, and integrate those projects into long-term goals.