Sina Fak is passionate about experimentation, and it shows. As the Head of Optimization & Business Intelligence at ConversionAdvocates, his team has been ranked as a Top 3 CRO Agency in 2020 by Clutch and works hard to turn that data into clear and compelling strategies that increase revenue, improves CX, and drive growth. As our guest on “From A to B,” he broke down the common missteps that occur in experimentation programs, and how to flip the switch to an approach that drives true change. He shares in his interview:
- Why experimentation is not a silver bullet
- How to balance establishing process vs. chasing results
- Why experiment goals need to go beyond validation
- How data can push a business forward
- Why your CRO manager needs additional resources
- How investing in data & research reaps rewards
Why can’t you guarantee that a given experiment will lift revenue?
Sina - “There's no silver plate or silver bullet that I'm coming to you with. The reason why the success that we've had and the clients that we work with have achieved success is because we come in with this curiosity, this beginner’s mindset. Every business, every industry, every product is unique. And our job is to figure out the intricacies,”
In large part, experimentation is about helping to reduce uncertainty. At times, it can help foster opportunities that companies are not even aware of.
Sina says that if an experimentation lead is always ready with perfect answers, there’s something missing. They are quickly going to reach the threshold of opportunity. In contrast, when you are open to all of the opportunities that lie with experimentation, there is no limit to the innovation it can bring.
How do you balance setting up a healthy process with the need to obtain and present results?
Sina - “When companies hire an experimentation person or a lead CRO manager, they're looking for solutions. They're looking for answers. They're looking for, ‘Hey, you must know the answers to our problems.’ The reality is - and this is something I'm definitely on a soapbox on - is companies don't even understand their problems. They don't have a truly deep understanding of what are they trying to solve before trying to solve it. They have this superficial KPI or goal. ‘We want to increase revenue. We want to increase conversion rates. We want to increase retention.’ But we don't understand what actually drives revenue or drives retention or conversion rates. We don't understand the intricacies,”
When you hire a CRO professional, that’s just one piece of the puzzle. You also need to have a clear grasp of the additional resources and tools that you need to accomplish your goals. If you don't have a clear idea of what your goals are, you won’t know where to properly invest in terms of the data and research that is needed to accomplish them.
What do you think of companies who want experimentation to validate what they are already doing?
Sina - “Data is this amazing thing where you can actually bend it and massage it into whatever you want. You can make data justify anything you want. And these fake data-driven companies are using data to actually not find truth, but just to validate their decisions, to justify the direction. They're never questioning the process that the data was acquired. They're never questioning the integrity of the data as it's being analyzed and manipulated. And a lot of the analysis that's being done—the segmentation, the manipulation— is for this storytelling. It's to make them look good, to save face. And it's done completely for all the wrong reasons,”
When you tout yourself as a data-driven company but you’re simply using data to make everything look good, that’s when you’ll never pass beyond vanity metrics. You’re more likely to pursue findings that gloss over the big picture in favor of individual metrics.
How would you characterize companies that use data to push the business forward, versus those that don’t?
Sina - “You have the completely opinion HiPPO-driven organization. Then you have the, okay, ‘we're collecting some data, but we don't really know how to use it.’ So we're data-informed, we understand it. And then you have the more data-driven. And that has so many facets to it. It's not only how you collect it. It's also how you integrate it throughout and connect the dots. You could be collecting a bunch of data in a whole bunch of different tools. But if you're integrating it and understanding how they connect and build on each other, there's a halo effect to data. You're stacking different insights,”
Stacking those data insights is obviously the ideal scenario—but it’s one that your CRO manager is not responsible for alone. The job of a CRO expert, in its most basic form, is to use data to ask questions. To accurately manipulate data, you need a data scientist or data engineer to get the proper infrastructure in place, not just a CRO manager in a silo.
What resources does a company need to truly maximize experimentation?
Sina - “It's crazy when I hear companies talk about ‘Hey, we have a CRO person!’ without any understanding of the structure and the support that needs to go around this person. It's like having a quarterback, but no linebackers. You're going to get demolished. And CRO experimentation…it just means using a scientific process to make things better. That's really what it all is at its core. But it's a multi-disciplinary team sport,”
The full team that you need will depend upon the company objectives. What are you trying to understand? What are you trying to optimize? What problems are you trying to solve?
Putting the burden of positive results on the shoulder of one person will never end well. In fact, it will in many cases breed a toxic environment where people may feel they need to outright lie in order to succeed.
Sina - “In that lens, that person will never feel comfortable to come to you with their failures. And failures have incredible potential for learning. You limit ideas and creativity and innovation, because I'm only going to go for those easy, very comfortable tests. I'm not going to drive any transformative, big ideas. Because I'm afraid to swing for the home runs because if I fail, I'm screwed. So not only does it limit that, but it also puts the CRO manager in a position where they're incentivized to lie.”
What's the biggest lesson you've learned in growing and maturing an experimentation program?
Sina - “When you start off with CRO, it's really easy to get those low-hanging fruit wins. And you can run experiments just based on heuristics, experience, some light research, and maybe a little bit of data that will get them results and get them wins. But you're going to hit a wall—a ceiling really,”
Sina - “If you are not investing in those two areas before developing your experimentation—hiring your copywriter, hiring your CRO manager, hiring your designer, project manager, developers—it doesn't matter. You are never ever going to maximize the value of that team if you don't have the research and the business analytics side of those pillars, that foundation, in place.”
You need to invest in your data and research team first and foremost, because that is what's going to drive everything else. For example, in Sina’s own work, ConversionAdvocates has been able to help clients improve messaging, introduce new product lines, and enter new markets.
That’s not the norm for experimentation. But they were able to succeed by partnering with teams like customer research, thanks to a shared bedrock of research and analytics.
Investing in that foundation will allow a company to go beyond A/B testing a landing page and innovate and improve from the bottom up.
.svg)
