Author: Martin P.
Title: Content Marketer
EP 30: User- or Session-based Test Analysis Blueprint
Money won't buy happiness, but it will pay the salaries of a large research staff to study the problem — Bill Vaughan
Yo yo, here we are, on another roll of Experimental Revolution.
To everyone who’s new, a warm welcome.
This is a place where we talk about what’s the latest in the world of experimentation.
Should you strictly follow P-value?
How can you reduce the noise with pre-experiment data?
What are the dimensions of the company’s tone of voice?
Let’s see…
This Week in Experimentation:
Blueprint of the week: User- or Session-based Test Analysis Blueprint — Formalize which mental model you’ll use when designing your tests. Link.
Talk of the week: How to Accelerate Innovation By Getting Results You Trust — Ronny Kohavi shares 3 principles you gotta follow if you want to trust 100% your test results. Link.
Read #1: The Four Dimensions of Tone of Voice — Explore 4 dimensions of the tone of voice that NNGroup turned into a framework you can use to impact customers. Link.
Read #2: CUPED Explained — see how you can treat almost statistically significant experiment results like they are with pre-experiment data. Link.
Read #3: 16 Ecommerce A/B Test Ideas Backed by UX Research — Speero has done like a lot of user experience on Ecomm. Here are 16 A/B test ideas you can try based on that data. Link.
Opinion of the week: I'm not a P-value absolutist. There, I said it. If there's a 'wrong/right' situation, a challenge is more perceived as an assault. — Link.
Event of the week: Measurecamp (Feb 11) — an unconference where speakers are fellow attendees and everyone is encouraged to discuss and participate. Link.
Blueprint of the Week: User- or Session-based Test Analysis
User vs Session-based test analysis - The blueprint tries to formalize the mental model you should have for using user counts or session counts while designing your test and deciding the metrics or performing the post-test analysis.
Use Cases:
— When setting up a study design, metric selection.
— When setting up metrics in behavioral tools.
Talk of the Week: How to Accelerate Innovation By Getting Results You Trust
Getting experimentation results isn’t easy, but it’s doable. However, can you trust your results?
From SRM to inadequately powered tests, all kinds of obstacles can corrupt the reliability of your experiment data.
Without the right guardrails, you risk basing decisions on wrong or insufficient data.
Kameleoon partnered with Ronny Kohavi to share 3 steps for accelerating innovation through trustworthy online experimentation:
1. Follow key statistics principles
2. Develop guardrails to check assumptions
3. Avoid test trustworthiness traps
Links to the video and full article.
Reads of the Week:
Read #1: The Four Dimensions of Tone of Voice
NNGroup envisions websites as tools that allow us to “talk” with our customers. If that’s true, a carefully considered tone of voice is critical for your marketing efforts.
The tone is more than the words you use. It’s how you communicate your personality. It’s how you tell your users how you FEEL about your message and impacts how they FEEL about your message as well.
But most advice about the tone of voice is vague — Be original. Be authentic. Be yourself.
So NNGroup developed a framework for describing the four dimensions of the tone of voice:
— Funny Vs Serious
— Formal Vs Casual
— Respectful Vs Irreverent
— Enthusiastic VS Matter-of-fact
They also ran qualitative usability tests and online surveys to measure the impact of those tone qualities on users, especially around credibility and trust.
Read #2: CUPED Explained
CUPED, or Controlled-experiment Using Pre-Experiment Data, is often described as one of the most powerful algorithms for raising the speed and accuracy of your experimentation program.
As your experiment nears completion, you’ll often see an outcome that seems only barely outside the range where we would treat it as statistically significant. For frequentists, this isn’t enough evidence for a change in user behavior.
For a real effect, you need more sample size.
But sample size costs a lot, doesn’t it?
CUPID tries to solve this problem by using pre-experiment data to describe away some noise in the test outcome.
Read #3: 16 Ecommerce A/B Test Ideas Backed by UX Research
Nothing works all the time on all sites. That’s why we test in the first place; to let the data tell us what is actually working.
That said, we have done quite a bit of user experience on ecommerce sites and have seen some trends in terms of what generates positive experiences from a customer perspective.
This post will outline 16 A/B test ideas based on that data.
Whether you’re just starting your first A/B tests or you’re just looking for continuous and never-ending improvements, these tactics are worth a look. At the very least, question your assumptions and give ‘em a test.
Opinion of the Week:
By Ben Labay
“I'm not a P-value absolutist. There, I said it.
Probabilities and confidence levels invite others to challenge in an open way. If there's a 'wrong/right' situation, a challenge is more perceived as an assault. Saying, I'm 80% sure this will beat control, implicitly invites challenge.
Science has in various ways standardized the expression of uncertainty to invite challenge. Science has also concluded that groups that are open to challenge function more efficiently, with less bias.
What does this mean for testing/CRO programs?
- plan tests (duration, sample sizes) using frequentist math
- call tests (after duration/plan) with Bayes math
- present uncertainty in probabilities, 'chance to beat control' terms.
Will this irk a stats geek? Sure, but everything does.
BTW, I think there is a path here to introducing "betting markets" to getting and integrating contrarian opinions within orgs.”
Event of the Week: MeasureCamp
MeasureCamp is an open, free-to-attend unconference, different to any other web analytics conference held around the world. Happening tomorrow on February 11, 2023.
MeasureCamp is an unconference, the schedule is created on the day and speakers are fellow attendees. Everyone is encouraged to discuss and participate in sessions, even to lead sessions themselves. You focus on what YOU find most interesting and useful.
We’ve come to the end of another email.
Thanks for reading it!
See you next week.
If you want me to share your podcast or blog post, just reply to this email or reach out to me personally at Martin@speero.com
PS — Think anyone could use all this info? Share the newsletter with them!
.svg)
