One thing: This is part 6 of of the Center of Excellence interview series between Speero’s CEO, Ben Labay, and a dozen exp. program managers from famous brands like Spotify, Booking(com), Vista, Hulu, AMEX, Disney, and more.
His goal? Research the complexity of scaling and structuring experimentation inside big and small tech companies. Check out the ungated, free Miro board with all the interviews and resources. Or listen/read to any interview from the series:
- How to build the right CoE with the Right Tech, Process, and Contract, with Stewart Ehoff, head of growth platforms and product operations at RS Group.
- Three Key Steps to Launching CoE, with Melanie Kyrklund, global head of experimentation at Specsavers.
- CoE: Change Starts Everywhere, with Ruben de Boer, lead experimentation consultant at Online Dialogue.
- Why One Structure Doesn't Fit All Companies, with Rommil Santiago, founder of Experiment Nation and Sr. Director of Product Experimentation at Constant Contact.
- How CoE can work like a Charm in a Brand with 6K Employees, with Luis Trindade, Principal Product Manager of Experimentation at Farfetch.
- Hitting the JIRA Wall–Why Waterfall Fails Experimentation and What Works Instead with Dan Layfield, director of product management at Diligent, ex-Uber.
- Centralized Teams Can Work Wonders in Small Businesses, with Kevin Anderson, Sr. Product Manager of Experimentation at Vista and the writer of the Experimental Mind newsletter.
This time, he sat with Dan Layfield, director of product management at Diligent. Dan also worked at Codecademy, a growth-stage startup, and at Uber, a company known for extreme scale and a mature product ownership structure.
Today, Dan is here to show how both startups and huge companies can scale with CoE.
Experimentation Evolution is Hard
Companies evolve. When they're small, with a handful of founders and engineers, decisions come from intuition. The founders know their market, they’ve done their research, and they push out products super fast, sometimes without a lot of precise tracking (why would they need it?).
But as a company grows, hits Series C or D, and scales to hundreds of people, the leaders shift. They go from being product leaders to organizational leaders. This means they need to build systems where they can truly trust their product owners on the front lines to make the right calls.
This shift is crucial, and it’s where we see a fork in the road for many organizations. Some build systems that foster this trust, embracing what’s often called a "product operating model."
Others stick to a more "project management mindset," where roadmaps are dictated from the top down, almost like an IT culture. This is where experimentation finds its place, and how it fits in varies wildly depending on the model.
The Product Operating Model: Autonomy Vs Accountability
At companies like Uber, experimentation is deeply embedded in the DNA. These organizations run on a "product operating model." Imagine "aligned autonomous pods" or squads. Each pod owns its own metrics and has the responsibility, authority, and resources to build its roadmap.
Dan told us how, in Uber, product managers are paired with engineering managers and designers, and they either have dedicated data science support or access to self-service tools.
Their job is to craft a roadmap that gives them the best chance to hit their goals.
In this setup, experimentation is a core tool for validating that they're moving in the right direction.
At Uber, pretty much "100% of things" were A/B tested. Everything from data infrastructure updates and things never visible to users all the way to front-end product changes. This shows how deeply integrated experimentation can be when a product operating model is in full swing.
This approach aligns well with Speero’s Test Phase Gate blueprint, which explores all the key steps you need to manage your experimentation program. It also allows you to determine the cadence and flow of your program, as well as understand the roles, activities, deliverables, and responsibilities within it.
Another blueprint connected to this is the A/B Testing Workflow Map, which lets you map out all the rules and steps for setting up, running, and analyzing the test. It also lists out steps, tasks, and decisions. For example, if something is not working, what is the fallback, etc.
The Project Management Mindset: Hitting the "JIRA Wall"
On the flip side, many organizations, especially older, larger ones, operate with more of a project management or IT culture. Here, a marketer might have a "crazy idea" and "shove it over a JIRA wall" to a service team that builds it.
The problem is, it’s a "waterfall type thing" – the idea is implemented before it’s truly tested or validated, making it hard to measure the real impact.
In these environments, experimentation often lives in a siloed team, seen more as a channel to drive revenue than a tool for continuous validation and learning.
The Center of Excellence as a Bridge
So, how do you bridge the gap between these two models? How do you move from a siloed, project-driven approach to an integrated, product-led experimentation culture?
Lots of orgs handle this by implementing a Center of Excellence (CoE). The idea is to elevate the experimentation function within the organization. A CoE aims to:
- Educate: Teach teams across the organization how to experiment effectively.
- Make tools accessible: Provide and standardize the necessary tools and platforms.
- Standardize processes: Ensure consistency in how experiments are designed, run, and analyzed.
This is a transition phase. Eventually, Dan says, the goal is to have an experimentation so universal that it turns into "a tool in the toolbox" for every product team, just like at Uber. But the CoE is the crucial first step to "spread it".
If you want to standardize and make your tools accessible, check out our free, ungated A/B Testing Tools Comparison page. Discover the perfect A/B testing tool for your experimentation needs and make the right choice for your organization without overpaying for tools.
Aligning Autonomous Teams: The Power of OKRs and Goal Trees
Even with autonomous pods, alignment is key. How do you ensure different teams are all pulling in the same direction? Dan explained that alignment happens in two places: "outside the pod and inside the pod".
Outside the pod: At the beginning of a quarter, each division chooses its OKRs (Objectives and Key Results). These OKRs "ladder up" to the broader organizational goals. There's a planning cycle where teams cross-check with each other to ensure alignment and anticipate any interdependencies.
Inside the pod: Within the pod, it’s up to the product manager, engineering manager, and designer to connect the dots for their engineers and designers. They show how their work impacts the numbers and the overall organization, whether through dedicated meetings, dashboards, stand-ups, or email reports.
This creates "skin in the game" for everyone, moving beyond just building code to truly solving problems. All the pods move in the same direction, while inside the pods people understand WHY they’re experimenting.
This system of alignment echoes Speero's Goal Tree Mapping blueprint. This blueprint helps visualize hierarchical goals, critical success factors, and necessary conditions. By mapping out metrics and sub-goals, it helps align CRO teams and ultimately the entire organization on common objectives. It also helps in defining strategic KPIs that fuel optimization efforts.
The What's in a Strategic Testing Roadmap blueprint further supports this by helping teams create research-based strategic roadmaps for testing programs and communicate objectives and key results from them.
Experimentation Never Stops
Dan’s experience highlights that learning experimentation is a "big jump forward" for anyone in product development. The core idea is to always try to connect "what they did and what it meant to the business".
While setting up the right tools, analytics, and understanding statistics are crucial, the foundational step is having clear targets and understanding if what you shipped impacts those targets.
Even if an organization can’t A/B test everything due to bandwidth, the mindset of trying to draw that cause-and-effect relationship is vital for continuous improvement and growth.
.svg)
