‍‍

‍Author: Paul Randall
Title: Senior Experimentation Strategist
Contact: Speak directly to author to discuss the topics

Briefly Experimental monthly aims to inform, educate and elevate the industry by sharing the latest in Experimentation learnings, and how these translate to better, faster businesses decision-making.

Edition 19, March

Key Take-Aways

• Growing experimentation teams; elevating the individuals already in your organization
• Experimentation Storytelling; telling effective stories is fundamental to the success of your work
• How to support new buyer habits with your B2B sales teams
• How do you qualify an "insight"? How much of the insights we share aren’t actually insightful (and how to fix it)
  

Headlines Of The Month

• Ruben de Boer takes a look at experimentation prioritization. This article will show you how to set up your evidence-based prioritization framework in Airtable, both in video and text. Check out the full article here.
• Chasing statistical ghosts; well worth watching and sharing to help improve you and your team's understanding of the pitfalls around experimentation analysis.

Dates For The Diary

• April 14th 6-7 pm GMT / 10am PT - The Speero And Intellimize Experimentation Maturity Webinar short-course: Speero has partnered with Intellimize to demystify Experimentation Maturity [XMAT]. Sign up to hear experts such as Ben Labay, Edgar Spongolts, Lennon Cole and Jocelyn Czerwinski from Intellimize and Speero discuss the 5 levels of experimentation maturity and clarify a) why it is important to progress through the levels, and b) what you need to do to grow from one level to the next. The discussion will also cover the 4 key pillars of experimentation and the roles each one plays in building a robust and optimized program. All webinar participants will receive a complimentary copy of the eBook covering all the learnings, guides, and insights. Don't miss it.

• April 27th [London, UK] - Experimentation Elite Conference and Award Ceremony: The UK's Only conference & awards dedicated to CRO & Experimentation. Ben Labay joins the Experimentation Elite Conference and Awards Ceremony, amongst other industry leaders and innovators, to speak about 'the value of experimentation innovation and scaling'. Whether passionate practitioners, CRO experts, in-house program leaders or agency innovators, the conference provides a highly tailored platform to bounce ideas, share winning tips and hear of the latest advancements in experimentation methodologies.    

‍

‍Experimentation Pillar 1 - Strategy & Culture:

‍‍THE BIG QUESTION : "How best to embed experimentation teams within larger organizations"

FOCUS: For many, a culture of experimentation is something to strive for; to extend beyond more traditional on-site testing into further wider-reaching areas of the business. Vistaprint’s parent company Vista understood the brief and created an Experimentation Hub. Let's look at how they did it and why they did it:‍

INSIGHT: Skills related to the experimentation process and analysis frameworks and development and deployment are housed centrally. At Vista they call this the Experimentation Hub. The key here is to understand the role of the Experimentation Ambassadors to be the link between the organization.

So what exactly are Experimentation Ambassadors...? If you want to build out experimentation from a central hub it would require a large amount of knowledge of all of these domains. A lot to ask of individuals when also dealing with stakeholders, and subject matter experts. When expanding into new areas, there will be new metrics, KPIs as well as skills and roles. If these skills are related to the domain being experimented in, eg: customer care or email marketing, these require local experts. Vista calls these individuals Experimentation Ambassadors.

CASE STUDY: The Vista case study shows how a culture of experimentation is used to scale. Vista works on a structured training program and mentorship to help these 'Experimentation Ambassadors'. The frameworks enables experimentation to flourish and be led by those embedded in localized teams, supported by a centralized practice.

It made me think of our SOPs that we use internally as an agency; you could consider Speero as the Experimentation Hub, and the clients we work with are our Ambassadors. It’s only possible by having this structured approach to enable us to scale to the point where our experiments can be centrally stored and accessed as an agency instead of scattered across multiple locations.

We’re looking at it both ways; embedding experimentation into organizations more by elevating those we work with and centralizing our approaches internally to streamline our processes. Easier said than done when you’re working with clients for nearly half a decade as well as onboarding new clients every year. Those systems need to constantly adapt to create consistency.

‍‍‍

Experimentation Pillar 2 - Process & Methodology:

THE BIG QUESTION: "Is it enough to do great work? or do we need to become better communicators, to allow for our great work to have the biggest impact..."

FOCUS: Experimentation Storytelling; At Speero, we’re no Pixar, but increasingly we've been looking at ways to best share the ‘story’ of an experiment. We ask questions like: ‘What are we trying to learn?’. How much of a gamble is it?’ ‘What would happen if the experiment does not win over the control?’

INSIGHT: Our test documents are our short stories. The nucleus of the idea is the hypothesis. From there we source insights and shape the purpose of the experiment. The scorecard is the summary, the conclusion or the abstract if you were to think of more scientific papers. It’s the one-pager you know will likely be passed to those less invested on a day-to-day basis.

Telling stories through research and experimentation helps create and magnify both context and relevance. With more context and relevance, the idea behind the experiment comes to life, making the subject matter far more interesting and engaging. Think of a teacher who made you love a particular subject. It wasn’t just what they taught, but the way they taught it. Our goal is to make experimentation fun, accessible and elevate everyone involved.

‍CASE STUDY: Reading about Pixar’s 22 Rules of Storytelling is a great example of an organization that is super focused on it’s approach. Here are a few of their rules and how they relate to experimentation storytelling.

7. Come up with your ending before you figure out your middle. Think of what your iteration of an experiment would look like? Where would you go if you learned the control beat the variation?

16. What are the stakes? A great pushback on simple experimentation is to say ‘what’s the worry of just doing this change? Leading to bolder ideas and moving faster.

17. No work is ever wasted? A learning experiment (one that didn’t beat control) is something we’re not afraid of. The naming is important too. We haven’t ‘lost’ anything. We’ve gained an insight and framing it in this way helps tell the story of why we may not want to go in that direction.

Here's another insightful read about Pixar's approach to storytelling and bringing personal experiences to life.
‍

Experimentation Pillar 3 - Data & Tools :

‍THE BIG QUESTION: "How do you support new buyer habits with your B2B sales teams?"‍

FOCUS: This Harvard Business Review article states “sales reps are no longer *the* channel to customers, but *a* channel to customers.”‍

INSIGHT: It may be no surprise that since the pandemic, shopping behaviors have changed dramatically, but what you may find interesting is that in a pre-pandemic study, B2B customer stakeholders only spent 17% of their total buying time interacting directly with supplier sales teams. This is the total time spent, so two or three vendors had to share that time with a single customer.

Increasingly there is a demand for independent learning online (27% of buying time) and independent learning offline (18%).

Consider your B2B journeys and consider if the amount of content available is large enough and concise enough for these individuals to spend nearly half of their time on their own.

When 43% of B2B buyers said that they would prefer a rep-free buying experience, the unlock here is in distributing as much relevant content as possible, but making it clear that experts are on hand.‍

CASE STUDY: SMART Technologies, a hardware and software provider to educators based in Canada decided to completely rethink their process. They dismantled traditional sales, marketing, success, and service teams and reconfigured them into a “Unified Commercial Engine”. Devoid of silos and handovers, these teams are built from mapping customers buying journeys. Based on the Jobs To Be Done approach, the five common buying jobs identified were Learn, Buy, Order/Install, Adopt, Support. Here teams are reassigned into these new teams. In 18 months, lead volume went up by 50%, lead acceptance increased by 35%. YoY growth stands at 48%.

This is a true example of taking a customer-centric approach and instead of making the customer jump through hoops, they identified ways to support them where most needed, leading to a much smoother process.

“Schedule a demo” has long been the B2B 'staple' of conversions. Reading the article on SMART Technologies, It's almost like the buyers don’t want a demo, they want answers to the questions they haven’t been able to answer themselves.

Instead of considering a funnel where scheduling a demo is where everyone goes, review your journey to understand how to help buyers serve themselves as easily as possible and understand at what point they need that direct interaction with a sales team.

‍

Experimentation Pillar 4 - People and Skills:

THE BIG QUESTION: "What exactly makes an insight an insight?"

FOCUS: Maybe it's easier to ask when is an insight not an insight? Avinash Kaushik has spent his life working helping people find insights. Whether it’s semantic satiation or an epiphany it’s clear that we don’t want to simply tell people what they already know (or could easily find themselves).

INSIGHT: ‍His new perspective is the "Out Of Sights".Things people can’t see. His explanation is a great one and touches on a compelling narrative; not just the what but the why.“the blue line is 20% above the red line because our biggest competitor launched a new product and priced it 10% below our best product.”CASE STUDY

With the bar raised, it enabled Avinash and his team to ask themselves if what they are wanting to share is really an out-of-sight piece of information. They broke it down into four attributes.

1. Novel: New and surprising
2. Actionable: Expressed with a clear implication for the audience
3. Credible: Data source - tool, people, entity - is respected by the audience
4. Relative: Expressed in context
   

But this could be overwhelming. Making people second-guess if a piece of information was truly out-of-sight. So they added Purpose.

The question-led approach gave them direction and the purpose to discover. Examples such as:

- What are the most productive inbound traffic streams, and which sources are we missing?

The first part could clearly be seen as an insight. The second part is the out-of-sight. The piece of data that wasn’t easily available but led to a more meaningful story to tell. Read his article in full here.

‍
----------------

FEEDBACK & COLLABORATION
Hope you have enjoyed this month's Briefly Experimental. If you have any feedback on the newsletter or would like to collaborate with Speero on thought-leadership content, please contact Morteza Maleki. See you next month.     ‍

‍
Join the monthly conversation with the leading minds in Experimentation by signing up to Briefly Experimental.