Siobhan Solberg, Founder of Raze, shared why that foundation comes down to measurement.
If you aren’t measuring and tracking, you won’t have the architectural plans in hand that tell you what to do next.
On From A to B, Siobhan explained:
- Focusing on the bigger picture rather than more tactical strategies to affect business change
- ‘Problem’ identification is extremely important - sometimes, moreso than solutions
- Tips on encouraging collaboration and buy-in for experimentation
What foundational tasks should people pay more attention to when building an experimentation program?
“It starts with really good measurement. We all know that the majority of analytics setups are faulty. They're going to be even more so now that we're switching to GA-4. It's not an out-of-the-box solution. It just isn't. It's marketed that way, but it's not. It's not just ‘let's put on tracking.’ You can look at so many websites, and it's all wrong. It's not that it's all wrong, but it's like duplicate page views or not tracking whatever. That's the first thing you need to get tight. How can you measure any kind of effect of any of the efforts you're putting in, let alone experimentation, without measuring first?” said Siobhan.
The truth of the matter is that not every company that comes to Siobhan’s team for help is ready for experimentation and testing. It takes a certain level of maturity, traffic, and revenue for experimentation to make sense for an organization.
Metrics build the core of a program’s foundation because tracking things properly takes time and requires a certain level of culture and buy-in within a company.
Why is having a solid research program and data to feed into your program so important?
“It's not just analytics, it's user testing. It's just QA-ing your site and making sure you don't have basic functional bugs. I mean, this is basic. It's your analytics, but it's more than that. Because at some point, our analytics is just collecting data points. You still need to create reports. You might need to model that data and include other data points from your Shopify account, from your Facebook, from anywhere else that you might be collecting it. Your POS system, who knows. You need to look at the bigger picture. And to be able to do that, you need the qualitative and quantitative data, and you need to be able to trust it,” said Siobhan.
Trust is the key concept here. If data is trustworthy, that means your entire company—not just your CRO team—can make informed decisions based on it.
- Your designer can confidently use user research data to design something that people actually want.
- Your marketing manager can use data to understand the big picture and how things are things working relative to one another.
- Your copywriter can understand if their copy is working well.
More importantly, the members of your organization will want to understand those things further, so that they allow the customer to guide their actions rather than their preconceived notions of expertise.
Why is it important to collectively agree on which business problems to solve?
“I feel like that is the foundation of the foundation. It's the architectural plan. It just doesn't work without it. If you're an agency and you're working with a client, this is what you're bringing to the client to say: this is what I think the problem is. Now I'm going to come up with some ideas. Depending on how hands-on they are. There have been plenty of times when the client themselves is like, oh, I have an idea. Or we have actually known this problem. And two years ago, we tested this with somebody else. It gives you more insight. And I think this is why working together in the culture is so great,” Siobhan said.
For one of Raze’s clients, Siobhan developed a competitive culture around tests. It wasn't a competition strictly in the sense of, “will the test win or lose.” It was focused on introducing the problem and challenging everyone to throw solutions into the hat.
“Do you know how many genius ideas they got this way? From different touchpoints, from customer support to the developer. The truth is the best ideas come from customer support. They are the closest to the users. It's free research. Those customer support logs are priceless,” said Siobhan.
That particular in-house team is seeing incredible success today, because they un-siloed their CRO employee and helped the entire company rally around and support them.
How do you promote collaboration in order to elevate your experimentation programs?
“As an agency owner, it's very hard to do this with clients, because very few times you have access. I think it is lucky because when I start the measurement process, meaning the foundation building, I do insist on a meeting with all the stakeholders. So I do get buy-in very early on. But they're not involved later, usually. There are some exceptions. But I think for in-house, this is really important,” said Siobhan.
Going back to the analogy of building a house, collaboration is like the plumbing system in the house. The water needs to get everywhere. The electricity needs to get everywhere.
A good CRO program doesn't just sit on one thing. To build a good experimentation program, you need to be experimenting with how to run that experimentation program. And not many programs take the initiative to experiment on themselves.
What KPIs can you use to test experimentation program success?
“They are hard metrics to measure. But it's like communication between teams. I would even measure how many requests you're getting from different teams. There are so many ways you can implement something. You can make it more mechanical, like people have to fill in a form. How many requests are you getting in that way? How many times are you being stopped at the water cooler? How many people are putting their idea on the public board that you threw a problem at? How many people are betting on the result? There are so many ways you can get people involved,” said Siobhan.
The key is that there has to be a sense of excitement about experimentation, and it has to come from the top.
Siobhan likes to get creative with how she can gamify experimentation in order to get people to care about the result. And if you present the result correctly, such as sharing it via a little report card, people start taking that information and thinking: How can this be applied somewhere else? How can we improve on this result?
What is the role of your test submission form at Raze?
“With all my clients, they get a link. And the link is ‘something@mytest’ usually. And I tell whoever hired me, usually the CMO or somebody, give this to your team. Let anyone who wants to add ideas. The more ideas we get from someone within the team [the better.] Or even let them submit problems that they see. It’s not because I'm going to use them all. It's because there needs to be this culture built up,” said Siobhan.
She uses the form not necessarily as a tool to generate ideas, but to grow the culture of experimentation within her clients’ organizations and get people on board the experimentation bandwagon.
“If you are an in-house person, I would definitely recommend [that you] open up every avenue. Don't be the person with the door closed. Don't be the person trying to shut everyone down. Go grab a beer. If people want to tell you over a beer what they think needs to be done, try to remember it. Take advantage of that, because this is going to be your gold mine,” Siobhan said.
The moral of the story: don’t sit back. Take action to get the data you need and build a successful experimentation program.
.svg)
