It’s important to know the most efficient way to arrange your optimization team to ensure their productivity and yours.

But what’s the best way to structure your team? Should optimization folks be in a separate team? Or under product teams? Or marketing?

There’s several different ways, and choosing which one works best for your company can be challenging. Here’s a rundown of the frameworks, their functions, and the benefits and challenges of each.

How can you structure your Experimentation team?

There are three types of optimization team models: centralized, decentralized, and center of excellence.

The centralized model is based on one team of employees, typically data scientists, and has the benefit of localizing expertise so that employees can develop a long-term optimization strategy, including developing better statistical algorithms and experimentation tools.

In decentralized (or “embedded”) teams, optimization responsibilities are distributed amongst employees under many different umbrellas within the organization, rather than one department.

The center of excellence model is basically a combination of the above two frameworks, utilizing both a centralized team and also distributing optimization employees throughout the different departmental branches.

How are central optimization teams be structured?

Chad Sanderson, the Experience Optimization Manager for Subway, creates an analogy for how the centralized model is set up:

The best optimization team mirrors the classical definition of a republic, or ‘a state in which supreme power is held by the people and their elected representatives, and which has an elected or nominated president.’

The Test Manager is the presidential figure in [this] scenario. They are responsible for guiding the direction of the program, facilitating the needs of various departments through focused optimization efforts, and perhaps most importantly, serving as a diplomat for CRO to the broader organization.

In the same way, the President is (or should be) surrounded by an experienced cabinet. So, too, should the Test Manager take direction from their cohort of subject matter experts: Product Managers, Technical Directors, Web Designers, and Customer Service Representatives.

Benefits of the centralized model

There’s a reason this is the most common form of optimization team organization: the benefits. From an organizational perspective, having all of your data clustered in one place makes it easier to monitor, reduces redundant testing, and streamlines developing a long-term strategy.

But data collection isn’t the only reason this model is used. There’s a significant employee benefit. In a centralized model, it’s a simpler matter to keep the big data picture in mind because there are shared resources and end goals for employees.

Spreading workers out over multiple departments can introduce conflicting motivations for whether or not an employee shares data. Centralizing can help you avoid competition and silos, or data and analytic walls that can prevent the sharing of pertinent data across teams.

Renee Thompson operates as a manager within TechTarget’s centralized model, and says that the localized nature of data helps her team plan for a long-term strategy:

In my experience, having a centralized function enables the institutional knowledge to be kept in one place, and makes fluid oversight possible, even if actual task work such as experimentation or research is conducted by multiple groups.

For example, it’s crucial that a systematic process is followed for testing, and the centralized function/team is in the best position to define that process and ensure that it is being followed for every project.

Centralized knowledge also enables organizations to learn from past research/tests, and to not repeat mistakes that may have been made.

Kristal Decoux agrees:
‍

I think it’s better to have a central team rather than being tied to any specific part of the organization. I’ve seen conversion teams nestled under marketing that resulted in product teams making their own testing tools, running unscientific tests, and making changes that didn’t align with marketing changes.

I’ve seen conversion teams organized by ‘outcome;’ a team optimizing for new customer acquisition, another for new product adoptions, etc. This results in competing priorities and a customer experience fractured by whether you’re new or an already existing customer.

The best configuration I have seen was when the conversion team was organized within a data and analytics organization. The optimizers keep close ties with the analysts and the BI team, making them VERY data driven. The organization doesn’t feel like they can only ask for tests if they’re in marketing (or acquisition, etc.) or another supported function.

This also allows the optimization team to be aware of the performance of the entire website and the business, rather than only what their functional role covers. Finally, this allows a strategic approach to optimization that includes the entire funnel. Imagine having one team optimizing for sign up, another for sign in, another for upselling/cross-merchandising, another for retention, another for engagement. The metrics aren’t the same and neither are the strategies…they all need to be aligned to create a consistent user experience.

Kristal Decoux, Amazon Web Services

Lastly, centralizing your efforts can deepen employees’ investment in the project outcomes Promotions, mentorships, etc., can be awarded if the employee is doing well; this is easier to track in centralized systems. This reward system can encourage more personal dedication.

Challenges of a centralized model

Employee investment in data outcomes is a double-edged sword.

When employees are rewarded for data that performs well, it can become tempting to skew the data for their own benefit. The objectivity of the data is lost. Fudge the data, and suddenly all of the statistics gained have lost all meaning.

Even if it’s just one metric, the veracity of everything else is immediately thrown into question.

Yu Guo, a Data Science Manager with Airbnb, agrees that there are certain benefits to using a centralized model, but it isn’t without its drawbacks:

Airbnb so far has a centralized data science and analytics function, organized into teams that focus on different product areas. There is a "foundation" team that builds the platform of experimentation, and internal clients optimize various parts of the business/product leveraging the platform.

One benefit is the ability to discuss and learn from colleagues, centralized planning for scalable process and tooling, and consistency in hiring, performance reviews, and career growth.

The trade-off is with better alignment of business objectives and resource/headcount planning. Collaboration with other functions is sometimes left to individual data scientists (or teams) to work out.

Leadership challenges and company culture can also pose obstacles for data scientists in a grouped centralized model.

Of course, data scientists will usually have to contend with some kind of HiPPO-style leadership at some point. In this case, it’s easy for bureaucracy to trump efficiency, especially in a company culture that doesn’t have a growth mindset. It's a challenge to get both leaders and data scientists in sync.

‍

As The Harvard Business Review writes:

[...] Data scientists may feel like outsiders when dealing with the businesses and thus be less attuned to the units’ goals and domain knowledge, which could make it harder for them to connect the dots and share relevant insights.

Moreover, the data scientists may lack the clout to persuade senior management to invest in building the necessary tools or to get corporate and business unit managers to trust the experiments’ results.

Example

This example from Optimizely gives a good look at the benefits of centralizing your data team, rather than having individual employees or consultants be in charge of optimization:

Imagine that you request an analytics report on a page you’d like to test. You want to know whether traffic and conversion rates are high enough to run a good experiment on the images on the page.

Any savvy analyst can pass you this data. But one who is invested in testing might notice an unusually high bounce rate and use of the search bar on this page; she realizes that visitors aren’t finding what they’re looking for.

Your optimization-focused analyst suggests that you test around this problem instead. She’s handed you an opportunity to make a bigger impact by fixing a glaring issue affecting your customers’ experience.

This type of cross-skills collaboration focused on testing makes your program much more efficient and more successful. An analyst who isn’t truly involved in testing and doesn’t understand the testing program’s objectives won’t apply his or her unique expertise in ways that allow for this.

How are decentralized optimization team models structured?

Embedded models can be set up several different ways, but it usually consists of spreading optimization responsibilities throughout different branches or departments like your marketing department, product branch, design team, etc.

This model tends to be more common in the SaaS service industry, in companies with a lot of developers, and other technical organizations.

Decentralized model benefits

There are advantages to this model, mostly that data scientists are forced to gain familiarity and expertise within multiple business branches.

Some organizations have a decentralized approach, which can work well when there are ‘experts’ across different business areas (particularly if they are in different geographical areas).”

Familiarity with a demographic or geographical area that your business serves can help inform your data collection and customer insight. It also gives a certain degree of employee freedom when it comes to work processes and flows.

Renee Thompson, TechTarget

‍

Decentralized model challenges

This model can also be messy. There’s significant opportunity for testing and analysis redundancy between teams, mostly due to silos and lack of communication between branches.

Reinventing the wheel is inefficient, and it’s important to avoid being, as Paul Magill put it, “a bunch of marketing islands, with our own language and ways of doing things.”

One drawback here is that it may be more difficult for the teams to learn from each other’s experiences, leading to redundant or duplicative work.
Some teams may follow a very formal and precise process, while others may “go rogue” with tests that don’t follow a process at all and result in business changes that may not be based on valid or reliable data.

Renee Thompson, TechTarget

On an individual level, decentralization poses some obstacles for employees, as the cross-functional nature of their positions can isolate them from mentoring and peer feedback.

Also, a company culture that that doesn’t value optimization can present big obstacles for data scientists. Matt Roach says it best: “Continuous optimization should be a philosophy, a way of working - not a consulting/audit engagement.”

As with the centralized model, efforts at convincing leadership to build experimentation tools can present a significant roadblock, especially without a number of other data scientists to back you up. Often, short-sighted leaders look for cheap quick-fixes, rather than long-term data and culture shifts.

No matter what model you have, ego will always be something you have to contend with, and not just with leadership.

Your optimization people need to be able to challenge ideas and commonly held beliefs and work at a different pace than other groups. If that is easier with a central team (and it commonly is), then that is how the team needs to be built.

Otherwise, the shared wisdom can help a lot (not all) of the organization. Either way, you need an independent and strong central leader, as there is going to be a never-ending contest between wills [...] People's egos are often far more important than what is actually best for the business.

You need to be able to take ideas from anywhere and quickly turn them into multiple options that challenge people’s beliefs and expectations.

As always, proving people wrong is far more valuable for the company then proving them right, but this, of course, means that egos are sacrificed for what is actually good for your users and for the company.

Once you are able to do that, you need to be doing it as often and as large a scale as possible. Downtime is the devil in terms of total impact to the organization.

Andrew Anderson, Recovery Brands

Example

Nick Meyers, Director of Design at FitBit, decided to decentralize his team when he noticed designers couldn’t expedite their product as quickly when they weren’t integrated with product teams. Coordination between departments was at an all-time low, and designers would quit as a result.

So rather than center the designers together on one centralized team, Meyers put one designer on each individual product team instead, paired with product managers and front-end engineers.

Though the transition wasn’t without its challenges--some designers felt isolated and pressured by engineers to value implementation over usability and user experience--the product development and design pace increased because communication between employees on the same project was smoother.

How is a center of excellence model structured?

The center of excellence model combines both centralization and decentralization principles, dedicating some employees to the central function of and dispersing the rest throughout separate departments.

Essentially, this framework establishes best practices and offers guidance for implementation of those practices. The central function develops tools and processes while leaving the collection to the individual branches.

Center of Excellence Benefits

This hybrid function instills processes and standards, but still allows for some creative wiggle room.

The centralized team is in charge of the design, execution, and analysis of controlled experiments, while the employees spread throughout the rest of the organization are in charge of the actual analytics.

Center of Excellence Challenges

Like everything, mixing models does have drawbacks. Often, employees are unsure which team (central or individual) owns which optimization responsibility, and which team receives the budget allocation for new hires when data requires team expansion.

Example

Sometimes, a blend of both models is best for your needs, and even big companies like Microsoft do this.

Paul Magill uses an example of a bank that was torn between the centralized and decentralized model, and finally developed a center of excellence model that worked for them:

They layered on strips of standardization: a unified language describing marketing job functions; a model of bank-wide career paths; communities of practice with designated “global conveners” that allowed marketing personnel from different units to take coordinated action (such as agreeing on a shared database or consolidating around particular market research partners);new governance bodies such as a brand council; key processes -- such as customer engagement and experience, and new market development -- that marketing should be running for the bank in all units.

As a result, the marketing organization was able to stop debating centralized vs. decentralized and instead position itself as a unifying force in the organization.

Teams moved away from debating whether, say, data analytics must be centralized or decentralized, and instead started asking questions such as “which specific activities in data analytics could benefit from being shared across units?”

The units have been able to increase the ROI of their marketing programs – mainly by increasing returns, in some cases by lowering investment – by calling on top quality pooled skills in shared centers of excellence.

How does the size of your company affect which model you choose?

It depends on your company’s resources. As Chad Sanderson explains below, if your company is small, you may want to choose the model that works best within your budget.‍

‍It’s important to keep in mind that the size and scope of CRO programs often have parallel, but not identical, operating models. A smaller team in a medium-sized business with less available data and resources must pick and choose their battles wisely.

Ideation, running usability tests, launching surveys, and analyzing web experiments is a lengthy process that necessitates strong hypotheses and higher than average win ratios to justify program ROI.

Larger businesses are faced with the opposite problem. It’s far more difficult to dramatically change the KPI’s of a well-known brand (which often number in the hundreds) and even more unlikely to correctly anticipate which tests result in major lift.

The most effective enterprise-level CRO programs have mastered testing at scale: Launching thousands of experiments per year to increase the likelihood of discovering wins. Program ROI isn’t as dependent on individual winning tests, but rather from sound experimental design and maximizing test velocity across multiple teams.

Chad Sanderson, Subway

Can an optimization team model evolve?

Yes. Ultimately, models are just guidelines for how to structure your company, and your model can change over time to serve the needs of your company as it grows.

Matt Roach, Sanoma: “Centralized (expert) teams can work well initially to come in and kick-start the optimization process, but after that, it must be executed by the resources in the business.”

Many companies start out with a centralized optimization team and then decentralize it as management sees fit. Yu Guo says even a large company like Airbnb isn’t opposed to changing its optimization model if the data and their needs dictate them to:
‍
Yu Guo, Airbnb: “[The centralized model] has worked well for Airbnb. In the future, there could be changes to how the team is organized, according to the changing business need of the company.”

Nothing is set in stone, and if someone tells you so, they’re bullshitting you. Much in the same way optimization employees are required to be flexible, so must your model be flexible.

As Ron Kohavi and Stefan Thomke writes:‍

In companies with multiple businesses, managers who consider testing a priority may not want to wait until corporate leaders develop a coordinated organizational approach; in those cases, a decentralized model might make sense, at least in the beginning.

And if online experimentation is a corporate priority, a company may want to build expertise and develop standards in a central unit before rolling them out in the business units.

Chad Sanderson expands on this concept, detailing how each model is best at different points in a company’s growth:‍

‍Different CRO program models work best at distinct stages of the team life-cycle. For example, if an optimization unit is in it's infancy they would likely operate most effectively underneath a larger department like Marketing, Analytics, or Technology.

Each win reflects positively on the department as a whole, meaning it would be far easier to gain resources for team members and tools. Stepping into the ring representing CRO isn’t nearly as hard when a top ranking senior executive is in your corner.

It’s also a great way for a fledgling program to find direction. Rather than jumping headfirst into completely changing the business culture, it might be necessary to first demonstrate how implementing CRO within a campaign results in meaningful incremental gains.

This not only cements the role of optimization within the company, but increases demand for your services across teams as well.

Sharing is the key to success

Data is useless if no one knows the numbers or how to interpret them. This is why silos are so problematic.

Renee Thompson, TechTarget, says;

Even with a centralized function, however, it’s important that input and ideas are coming from across the business from the folks who are closest to the customers, data, etc [...]

No matter which approach is used, always ensure things are being documented and shared. Sometimes even in a centralized group, we forget what we’ve done in the past or forget the results of specific testing/research!

If you have a centralized function, there’s always someone to go to with questions or problems. With a decentralized team, it’s important for someone within each group to be responsible for documentation and interaction with the other groups. Communication is key!

Your company is working against its own self-interest if accumulated data isn’t being shared across all available platforms to the employees the information is relevant to. Sharing provides clarity and accountability.

Big picture: don’t get too hung up on the model

Models are important, sure. Whatever you choose has to work best for your company, though, and tweaking the framework to serve your needs is relevant than following the model to a T.

Andrew Anderson, Recovery Brands:

It is less about where [employees] sit [and more about] how they think [...]

Optimization is about maximizing efforts and actions by testing as often and as many different options as possible. Doing just two experiences or just testing when a new idea hits the table cripples the value of the efforts [...]

Focusing on just the structure of team misses the key things that actually determine success and failure for optimization.

Focus on the things that matter, discipline, proving people wrong, constant action and tackling everywhere not just the shiny object of the moment, and the team format and how that fits into your [organization] will become clear. Failure to do that will limit your program no matter what and how you structure the team.

At the end of the day, every formatting decision you make is a balancing act. At the end of the day, you have to figure out what’s right for your company.

Conclusion

There are three types of optimization team models currently in usage: centralized, decentralized, and center of excellence. Centralized is the one most commonly chosen in the business world, but each has its advantages and disadvantages. Finding the balance between employee needs and business requirements is key to developing a working system. To figure out which one to implement, you must know:

• Your company size
• Your optimization budget
• Your main-focus metrics
• Your types of optimization employees
• How your employees communicate best with each other

Once you establish these five points, you’ll be able to select which of the three frameworks is best within your company. However, it’s important to remember that the model isn’t as important as the availability of data: make sure your data is available to all relevant parties across all platforms.