Why do you experiment? Why should we experiment? Why should the organization experiment?
I think expanding our understanding of the wider purpose of testing, can hugely impact the effectiveness of tests. And so resulting in better performance and positive change along the way.
We often speak of ‘storytelling’ in the context of marketing, and its effectiveness in creating engagement; but how can Experimentation and CRO leverage storytelling? Where are the stories in our field right now? What are we telling ourselves about experimentation as a mechanism for growth?
The first step in storytelling is a unified narrative: I’ve worked with our marketing manager Morteza Maleki on helping me redefine these sets of narratives for our industry:
- Is data truly being understood?
- How efficient are our leadership decision-making models?
- Why such focus on results without understanding process?
These are just a few discussions which now need to be considered in the industry’s evolving story. These areas help advance the applications of testing and the science applied to business generally.
The new narratives for the experimentation industry:
1. 'decision support' (the point of it all right?)
The cost of decision-making inefficiencies
> Businesses are a series of people making decisions, resulting in processes and outputs. How is the efficiency of decision-making directly linked to the success of the business?
Decision Intelligence Protocols and developing Business DiQ
> How to use experimentation programs to advance your businesses decision making efficiencies [DiQ]; the ability to make better decisions, faster?
Holistic Experimentation; company wide Exp Proficiency, Agility, Community for Decision Intelligence
> How do orgs create DiQ via operational process standards?
2. What stories are we in the industry telling about DATA right now?
https://www.linkedin.com/posts/ben-labay_experimentation-cro-activity-6886676333615808512-d2cf
Data is a huge narrative for the CRO industry, and probably the most important current narrative IMO. Data is the core ingredient for a decision. The better the data, the better the decision.
Here are two core 'narratives' related to data that I see:
What is data efficacy
> As we shift from the race to collect data, to the race to best understand and action that data, our data processing infrastructure becomes crucial to success. So where are the gaps and how can we develop robust ‘rituals’ to better leverage the data for business decisions.
Changing perspective on data for decision support (with EXP)
> What if EXP data was a product? What are the expectations of this product? How do you maintain this product? How do you continue to improve this product. (I see 'data-warehouse as the new backend' being a part of this narrative)
Chad Sanderson Lukas Vermeer Craig Sullivan Ronny Kohavi what am I missing? What other core stories/narratives/trends are we seeing with data right now?
I am purposely skipping over 'statistics' as a core narrative as I think it's implied, always in the background at this point. It's the furniture in the scene that we're creating with the other stories.
3. How experimentation helps orgs with innovation and change:
Experimentation and the Speed Of Innovation
> Creating confidence in product dev and innovation, with better faster decision making
Digital Transformation Investment without the research, data and experimentation culture to support it
> The illusion of being ready for digital transformation
Role of EXP in experience vs. role of EXP in business and product strategy
> The value of Exp beyond customer experience and conversion
This set of narratives speaks to a group of folks that are pushing for experimentation to be more a core part of a business operating systems. Manuel Da Costa is pushing a lot on this, and we see top tech firms like Netflix, Microsoft, Booking, Uber have testing in their DNA...sometimes to the degree that EXP 'centers of excellence' aren't even needed.
Search 'CRO' vs 'Experimentation' in Linkedin, under the 'people' category. In the first you'll see people with roles in marketing, in the second they will be on data and product teams. This narrative set above speaks to the need to bridge these worlds.
4. The big ‘Why’ we experiment
Why do we experiment? What would the company do otherwise? What are the stakes? How do companies that experiment win?
I've created two 'strategic narratives' using a rough version of Andy Raskin's framework for strategic narratives:
1. The new world
2. The stakes and costs in this new world
3. The new ‘game’ to play, with winners and losers
4. The cheat code, a magic capability we should know about
5. And the proof?
My two narratives have two audiences:
1. CMO/CPO - this is the PRACTITIONER narrative, and one that I'm sure most people reading this will be familiar with. It preaches to the choir. It speaks to the flood of (sh!tty) data at hand and the need to make decisions quickly. It speaks to the democratization of the randomized controlled trial and science principles generally. The narrative here isn't new, but I do like to see it broken out like this, it makes it accessible and wieldable in convos.
2. CEO - this is the RESOURCER narrative. This is new, I don't see people talking about this but it's a HUGE hidden narrative. In small orgs, and even 'small' 100mil ecom orgs, the CEO is also the practitioner...so this narrative doesn't apply there. This narrative is for larger, legacy companies (think any company that existed before the internet and still might have a fax machine). Large and mid-sized enterprises. They need innovation and digital transformation to stay relevant. They need to attract modern talent that wants autonomy in work and needs an operational system (experimentation) that pushes decision making down corporate ladders. This strategic narrative speaks to the need for testing and decision systems to promote the workplace, to attract better talent. Do you want to work for Walmart or Walmart Labs??
Narrative #1 above has driven CRO growth and industry $ for the past 10 years. This $ will be eclipsed by the potential in narrative #2.
Think about what the consultative and change management agencies/firms like Accenture/McKenzie/Bain have been doing and charging. But now get tactical with proper data management and decision systems based on science. This new narrative will dominate a small group of agencies working on the bridge between digital transformation and practical data and CRO work.
.svg)
