Experimentation is a multidisciplinary field where we help businesses make better decisions. In 2016 CXL pulled together a list of top traits that an ‘optimiser’ needs in order to be “great'' rather than simply “good” at website optimisation. Six years later, here at Speero we thought it’s time to revisit these traits to see if and how these have evolved.
In 2016 we were mostly talking about CRO and how CRO people needed to be multi-talents and amazing at analysis, copywriting, design, consumer psychology, writing java-script, QA’ing tests, project managing, business strategy etc Basically technical, creative, analytical, entrepreneurial and very strong in humanities.
Now in 2022 we are seeing a shift towards more companies and agencies opting for teams of experts in different areas. This doesn’t mean that “an optimiser”, “an experimenter” or “a CRO person” would no longer exist. Of course they are still needed but they would now be working in a team of experts, pulling everyone’s knowledge together in order to run an Experimentation Operating System; teams that create business-strategy-based roadmaps for better customer experience, decision making and website optimisation.
CXL wrote 6 years ago that optimisers benefit the most from being analytical, user and data driven, Kaizen (always trying to improve), process oriented, curious, detail oriented, humble, and understanding design. Apparently, the first 5 were mentioned by everyone in the industry who gave their view.
For an updated list, Speero put together a number of traits from the previous study but also new traits that were mentioned on LinkedIn or that the Speero team brainstormed. Then everyone prioritized their top 5 and we also asked the industry to add their top 5 to our scoreboard.
After gathering all the inputs we saw quite a few old favorites being celebrated again like analytical thinking and curiosity but there were newcomers like teamwork, being entrepreneurial, showing passion etc and the priority has changed as well.
The new prioritized list of the most important traits that an experimenter should have or showcase are as follows:
1. Curiosity, 2, Being a team player, 3. Humility, 4. Critical thinking, 5. Analytical thinking, 6. Being data-driven, 7. Proactiveness, 8. Having an entrepreneurial mindset ... and a bonus trait - Kaizen - always trying to improve. But what do we actually mean by these traits and why are they important for an experimenter?
1. Curiosity
Already in 2016 Oli Gardner pointed out the importance of this: “The single most impactful tool for CRO is curiosity.” Experimentation expert and practitioner Lorenzo Carreri said:
“One of the most important skills that you should develop as a CRO / Experimentation professional is: The Art of Asking Questions. You can have average user research, analytics, UX, etc skills but if you are obsessed with asking yourself questions, you'll be ahead of your peers. Curiosity is what fuels CRO people.”
The list of questions that Experimenters should be asking every day is endless but it often starts with “why” and expands from that. Why are we seeing these results, what else could help us to understand the results, what are our customers’ needs and what is the best way to satisfy them?
Bobby Hewitt, founder of Creative Thirst, agreed, saying, “Without an insatiable curiosity that demands more, you won’t get to the root.”
“Curiosity – wanting to know how everything works – how the analytics data is collected, what a bounce rate really means, what the reports are actually saying from an atomic layer up. Someone who will not give up because they’re interested, they want to find out or they absolutely have to know the truth, how it works and what it all means.” - Craig Sullivan
2. Team player
As the American Psychological Association (APA) is putting it: “The lone wolf is becoming an endangered species. In fields from health care to hospitality, startups to big business, teamwork has become the favored way to get things done.” According to Susan McDaniel, PhD, a psychologist who is known for her dedication to team-based work said that the world is so complex, no one person has the skills or knowledge to accomplish all that we want to accomplish.
This is what we’re seeing in the Experimentation field as well, we are creating teams of people with different skills rather than expecting one person to be an expert in everything. The standard of the work will be much higher and the amount of work is also increasing so it’s not viable for just one person to do the majority of heavy lifting.
Tracy Middleton from Atlassian wrote about the benefits of teamwork and how it has become even more important during and after the pandemic. She brought out 10 benefits with direct implications to the company’s bottom line but her top 3 were the following:
- Research shows that collaborative problem solving leads to better outcomes.
- People are more likely to take calculated risks that lead to innovation if they have the support of a team behind them.
- Working in a team encourages personal growth, increases job satisfaction, and reduces stress.
3. Humble
There’s no room for egos in optimization. You’ll find that many CRO articles lambast the HiPPO, and for good reason: opinions, no matter how highly paid, aren’t worth much compared to humility and data. As Joel Klettke put it, “there can be no egos in CRO – anything can change, and you can’t get married to one element or angle, because it could be the wrong one.”
Oli Gardner continued on that point, saying: “It can be really hard to let something go when you’ve sweated over it…If it loses, you have to have the courage to throw it away. The best way to do that is to celebrate the fact that you learned something from the failure.” And that’s where the lean startup mentality comes into play. You have to learn to love failure, in the sense that you’re learning something. Better than continuing on a disillusioned path of bias and dissonance at least.
Alex Harris:
“Many CRO experts have years of experience, but we are all still learning everyday. When you think you know everything about executing successful tests, even the most experienced optimizers will be surprised when a hypothesis fails. Realize this cycle of trial and error is a never ending cycle of knowledge. Every website, project or experiment is unique and the best results from surrounding yourself with people smarter than you.”
Craig Sullivan:
“So next time you hire a UX or CRO practitioner, go for the one who shows humility — who may not know from looking at your company’s engine with one glance what’s wrong — but who will roll up their sleeves and find out, using every tool at their disposal to find truth, solutions and the desired outcome.”
4. Critical thinking
People with critical thinking are able to understand logical connections between ideas. They have an open mind, think outside the box and are able to retrieve different information and form them into new ideas. (UKEssays)
A major part of experimentation is about problem solving. You’ll always have a slightly different set of data and problems you need to work off. Without critical thinking it is impossible to come up with logical creative solutions.
I asked my colleague at Speero about what makes him a great Strategist rather than good. He said:
“The 'winning' part I don't really care as much about, necessarily. It's understanding the business, then identifying ways the tool (experimentation) can help them (the client). But most of the time, the people I'm trying to help don't know about experimentation the way I do. So I get to educate on experimentation, but also get creative based on their limitations and the information they need to know. Winning is the bonus of laying down the experimentation process well enough. It's problem solving to problem solve.” - Shiva Manjunath
Critical thinking is one of the key skills that Speero assesses Strategist candidates for. We task people to send us an A/B test brief of a test they have previously run and are proud of. That brief reveals how well people make connections between insights, problem statements and potential solutions and also how they interpret the results and come up with the recommendations for next steps. These links between the process steps are important and they need to be logical.
5. Analytical thinking
“The number of CROs available however, is so small that it’s very difficult to find talent, but it’s also harder to try and start from scratch. If you were going to hire someone less experienced, I’d start with someone with an analytical mindset.” - Oli Gardner
Peep Laja has said that good optimizers are a friend of numbers. We are still seeing people holding themselves back and not letting themselves run free in the numbers world or analytics. There’s a level of cautiousness and intimidation. The people who can break that mental barrier do well. This doesn’t mean that you have to be a data analyst or a total math nerd but you need to understand what to take from numbers, where to investigate further and what to take with a pinch of salt. This is easily learnable but takes courage and time to get comfortable.
These days it is also advisable to hire a separate data analyst who can take the initial findings further, build models and delve into customer reports understanding all the different marketing tools your company or client has in order to understand potential impact of the A/B tests but also find data to support decision making and ideation for iterations.
In my experience the strength of a data scientist is pulling out as accurate information as possible, but it’s the Experimenters job to interpret it and come up with insights, recommendations and next steps from the data and put them into wider context.
In addition to analytics, an optimizer needs to have good command of statistics. They have to know enough to understand statistical significance and statistical power, probability, the importance of sample sizes and the representativeness of the samples, understand data pollution and so on.
Summary
There are many general traits that are beneficial in order to be successful in a professional career but there are some that are more important and specific to Experimenters today. It is still very customer centric and process oriented although they have become so second nature that the industry doesn’t necessarily call them out. One of the biggest shifts we are seeing is the importance of team work and everyone pulling their own weight but also contributing with their area of expertise.
.svg)
