For a lot of companies, experimentation is a technical solution. You get software that lets people run experiments, and that's it. But that's a narrow view.
An experimentation operating system is how the business organizes itself around experimentation:
- It includes the systems that allow collaboration and knowledge sharing.
- It’s about the rituals and agreed-upon ways of working.
- It's about aligning experimentation with strategic goals to drive innovation.
Expedia, for example, is a massive enterprise. And for them, a big challenge is that they have a lot of unique teams and silos. Each team has its own goals, key results, and customers. They are a two-sided marketplace, with some teams working with partners and others with travelers.
This means a one-size-fits-all solution just doesn't work. They can't just slap a product model on the entire organization. This is where you need to get smart.
Welcome to the second Speero Sideshow. It’s a podcast on why we need to start talking about growth experimentation. Because experimentation can be a holistic driver of innovation and growth in every business area.
We’re bringing a new operator, strategist, or experimentation/research/growth leader every week. This time, Paul Davidson from Expedia joined us to talk how experimentation isn’t a method of discovery or validation, but an operating system.
TLDR:
- Experimentation should be viewed as a full organizational "operating system," not just a technical tool for A/B testing.
- The motivation for experimentation can be about generating revenue (a CPO/CMO goal) or about building a culture that attracts top talent (a CEO goal).
- Your company's org chart heavily influences how you can scale experimentation; a rigid structure with siloed teams makes it harder to innovate.
- A major challenge for large companies is knowledge management—ensuring that insights from past experiments aren't lost and are shared across the organization.
- AI is a powerful new "Trojan horse" that can help flatten roles and enable bigger, more complex experiments by making it easier to surface insights and streamline processes.
- Driving organizational change requires a balance of top-down goals from leadership and bottom-up evangelism from empowered teams.
- The real competitive advantage comes from building a cohesive system of people, processes, and technology that can learn and adapt quickly.
The Two Strategic Narratives of Experimentation
There are two main reasons an organization might adopt experimentation. One is the classic reason: more acquisition, more monetization, more retention.
This comes from the CPO or CMO and is focused on customer experience and revenue goals. The idea is that experimentation will lead to better decisions and, ultimately, more money.
But there’s a second, culture-based narrative. This one comes from the CEO and is about attracting the right people. A culture of experimentation attracts people who want to work in a non-traditional, top-down way.
They want to be able to try things and learn, not just execute a roadmap. Some companies even have the explicit goal of becoming an experimentation culture to attract this kind of talent.
This is an area where the Speero How to Engage Community and Create a Culture Around Experimentation Blueprint can really help. It offers ways to get buy-in and educate your teams on testing principles, which is crucial for building a culture that attracts and retains top talent.
But before you can even get to a culture of experimentation, you have to iron out your product operating model. In many companies, stakeholders think the IT team should just do what they're told, while the product team sees itself as responsible for the product and wants to validate ideas.
This misalignment of mental models gets in the way of everything. This is something Expedia deals with constantly.
As a publicly traded company, there's a constant push for short-term results. They have to balance innovation and learning with showing progress to leadership and Wall Street. This is a never-ending push and pull.
Your Org Chart Is How You Ship
The way you organize your company has a huge effect on how you scale experimentation. Ben brought up a good point: you ship through your org chart. Is your team throwing a Jira ticket over a wall, or do you have an empowered team with a developer in your squad?
There are a few different org models. A functional organization (like Apple) has functional excellence, where everything like design goes through one team. This leads to uniformity across products.
A general management or business unit model (like Amazon) has business units that are built on their own, with their own stack and workflows. This allows for more experimentation, but it can be less uniform.
Expedia is a bit of both. They have "verticals" that are similar to business units and "horizontals" that deliver core site functionality. They also have different brands and different products, like hotels and airfare, which adds another dimension of complexity.
This means that while teams are empowered to drive their own goals, it's very difficult to align a big tech initiative across the entire org. The problem is that experimentation is often seen as an afterthought.
It's a technical thing you do after you've built something to make sure it works. But this is a "cart before the horse" situation. Once a squad has invested three weeks in building something, they don't want it to fail because that means they wasted resources.
This often leads to teams looking for a win in a small segment, even if the test failed overall. This is a huge problem that can be solved with a more rigorous approach to product discovery and validation.
The Knowledge Management Problem
Scaling a learning organization is a huge challenge for enterprises. You want to be able to build on past learnings and not re-test something that was already done a year ago. But most companies struggle with connecting the dots.
Expedia is still early in this journey. Historically, experiments were treated as siloed events. Learnings get shared with the immediate team, but then they get piled away. They have a lot of tools that capture bits and pieces of information, and the financial data is stored in different places.
What's missing is the connection from the C-suite strategy down to the individual experiment, with all the insights and observations reporting cleanly back up the chain.
This is the exact problem that the Speero Test Phase Gate Blueprint addresses. This blueprint helps you manage your experimentation program by defining the phases, stages, and process itself. It helps you ask the right questions at each stage, ensuring that everything is aligned and that learnings are documented and shared.
AI: The Trojan Horse on Steroids
AI is going to change a lot in experimentation. On one hand, it's an experiment itself. On the other hand, it can help the experimentation process. You can use AI to help surface insights from old experiments, which is something a human would find really hard to do.
But you have to be careful not to use AI as a band-aid for bad processes. Paul gave a great example: they have a product requirement document (PRD) template, and they could use AI to scan it and fill in the experiment for them.
The problem is that it would just be a band-aid, and the results wouldn't be great. He'd rather have the PRD become a standardized document with metadata fields that explicitly fill in the experiment setup. That way, the process is structured, and the AI is helping, not just implying what the experiment should be.
Experimentation has always been a Trojan horse for changing how a business works. It's a tool that gives people on the front lines the data they need to be autonomous and accountable. AI is just this idea on steroids.
It's going to flatten roles and org charts, and it's going to allow us to think bigger. We can now put on our experimenter's cap in areas we never thought possible before. It's a forcing function for that conversation to happen.
Top-Down Goals and Bottom-Up Evangelism
Driving change in a big organization is a challenge. You need a balance of top-down and bottom-up approaches. In a large organization, you feel a lot of pressure from the top down to deliver results. If you're not aligned with those goals, you're going to struggle.
Expedia has a lot of champions for experimentation, but they don't always feel empowered to act. They need to be able to demonstrate how experimentation helps them achieve their goals.
This is where the top-down pressure meets the bottom-up evangelism. The leaders need to push the values and the people on the ground need to show how they're winning. Then, hopefully, other teams will adopt those successful practices. This is how you drive growth.
This is also a core part of the Speero Program Metrics Blueprint. It helps you track the success of your experimentation program by monitoring things like test velocity and effectiveness.
This gives you the data you need to identify bottlenecks and show how your team is making progress. It helps you speak the same language as leadership and demonstrate the value of your work.
The reality is that it's a constant push and pull. Experimenters can get too focused on the science and forget the applied science and why everyone is working together. The key is to have a healthy, transparent dialogue where everyone understands the give and take.
Ultimately, using concrete examples of what worked and what didn't is the best way to persuade people and get buy-in. This is a great way to start building a playbook that other teams can follow.
Don't Win, Create A Winning System
The true challenge of experimentation isn't finding the next winning test; it's building a cohesive organizational system that can adapt, learn, and grow faster than the competition. This system, which connects people, processes, and technology, is the real competitive advantage.
To continue this journey, start by taking a critical look at your own organization's processes. Ask yourself if your team's current structure and methods are truly supporting a culture of learning, or if they're holding it back.
Find out where your company's silos exist, and talk with other teams about how you can collaborate to share insights and build on past learnings.
Finally, consider how you can use concrete examples of both successes and failures to persuade leadership and other teams to adopt a more strategic approach to experimentation.
This will help you build a playbook that everyone can follow, ensuring that your program's lessons are never lost.
.svg)
