Can you optimize our site? What uplift can we expect? Will this test win?
CRO is often misunderstood. But if it’s not about winning, what is it about? On From A to B, Shiva was joined by Tim Mehta, conversion optimization manager at Root Insurance.
They dug into:
- Why experimentation is like the stock market
- How to set realistic expectations within your team
- What language speaks best to leadership
Why is the role of CRO commonly misunderstood?
“I think it all started with whoever coined it as conversion rate optimization, and then they kind of screwed all of us. If you think about what marketers do, we love to ruin things. And so as that particular phrase was coined, it stuck. Because it creates a very attractive promise. And in reality, we do a lot more. Conversion optimization really is about 20% of our job, because about 10-20% of our tests will actually optimize the conversion rate that you're looking for,” said Tim.
Tim likes to explain to others that CROs are experimentation managers. CRO is essentially an R&D department whose main product are insights and learnings that can lead to better decision making. We’d be better served by changing the name from CRO to DMO: decision-making optimization.
CRO goes far beyond landing page conversion rate optimization.
“I try to compare it more so to the stock market, as far as the performance you should expect out of it. If this [someone] would ask a financial advisor: what's the one stock I should choose that's going to make me rich? They're going to give you the same answer that we should be giving which is, well, it's actually your portfolio of ideas and tests that we run. That's where you should really focus your expectations and your level of performance,” Tim said.
When you emphasize only select ideas and goals and put high emotional investment and expectations on those areas, you’ll set yourself up for disappointment. The real value experimentation brings is in the whole portfolio of experiments that you've run and the insights you've gained from both wins and losses.
How can you expand client goals beyond projected lift?
“I think it's important to emphasize the low cost of testing in order to get past that particular speed bump. Because they're looking at it as, ‘Hey, what's the expected lift of this individual test, so we can first know if we should invest resources into building it in the first place?’ They're asking for what the outcome you're looking to obtain is before you actually do it. You're dealing with a similar but different mental model. The mental model being: we're used to giving projections in business and marketing all the time. The funny thing is that experimentation is not one of those areas where you can give very accurate projections. You can give extrapolations based on past tests of what you could expect to make. But as far as individual tests level, we can't project what our average impact or what our expected impact is going to be,” said Tim.
Trying to provide that type of projection would look like a graph chart with arrows going in every single direction.
Beyond reviewing past data from similar tests or qualitative data that you have access to, it's impossible to make predictions of that nature. That’s why organizations should have very holistic expectations for their testing programs.
How can you approach expectation-setting within a team?
“Companies who are not familiar with the practice itself, they want someone who's reasonably priced who can do everything that's needed for experimentation to work. So first, they get the job description for all the line items you need for experimentation. And then they put it out there for one role. And they're looking for a unicorn. If they are going to find someone who can do all those things, they're going to cost a lot more than what you're expecting. Or they're just going to tell you: I can't do all this,” Tim said.
Established companies with serious goals for experimentation need to accept that they need more people than can be covered by one unrealistic job description.
Early companies who are just trying to get their feet wet have another avenue available to them, which is hiring an outside agency. This can be a great first step for companies that don't know how to invest in it yet and want to spend a year or two observing and understanding the process from others.
The bottom line is you need to determine what your risk tolerance is, how much you can invest, and where.
Why do you need leadership to embrace experimentation?
“The leadership culture of experimentation—it's also one of those practices where it doesn't get prioritized from that resource perspective, because you don't have that culture of innovation at the top. I think that's one of the most important things. If you're just in a silo and you hire your own CRO practice, then you've got down-bottom selling you need to do for something you're already investing in. And it's a much steeper climb, because you have to keep proving it out at every stage,” said Tim.
If you have a person who can come in and immediately help leadership understand the value of experimentation, you’ll be in good shape. All successful companies today have been doing some type of experimentation to get the success they have. They just might not label those efforts as “experimentation” explicitly.
It’s invaluable to have an in-house person or an agency who can help smooth internal politicking and be a change manager for your organization’s experimentation efforts.
How can a CRO get other team members excited?
“You have to really understand how to make people care, and it changes depending on whoever you're talking to. Developers care about saving time. Designers care about insights that can inform their designs. Business cares about how much you're saving, how much you're making. Analytics cares about the integrity of the data, the different methods we're using, making sure we're being very stringent. So you have to speak all these languages, but you have to do it in a way that says: experimentation brings value to what you care about, and here's how,” Tim said.
When you can demonstrate visibly how experimentation efforts can impact those specific areas, you gain people’s interest, investment, and trust. The more you can do to generate shared excitement about results and minimize resistance, the more successful you will be.
How do you communicate that the value of experimentation goes beyond winning?
“Communicate what's important to leadership. Yes, winning and showing our potential money earned is really important. But I think another thing that everyone should be doing to start speaking that language as far as what's important to them is operational efficiencies. So if you're early on in a program that is just brand new, you have maybe your first 2-3 months to establish a baseline, and then the next three months to see how much can we improve that baseline operationally,” said Tim.
Improving the operational baseline means adding efficiency by asking things like:
- How many more tests can we run in that short timeframe?
- How much faster can we do analysis through automated reporting?
- How much faster can we collect requirements through a JIRA ticket so we have the metadata we need?
There are many opportunities for time cost savings within an experimentation program, and those will speak clearly to leadership.
Saving literal money and saving “time money” are both going to impact an organization’s bottom line.
.svg)
