Author: Martin P.
Title: Content Marketer
EP 33: Should I Run an AB Test?
If you steal from one author it's plagiarism; if you steal from many… it's research — Wilson Mizner
Hi there.
Martin P. here.
The weather’s getting better, at least in my part of the world.
Easier to work when it's bright outside.
Easier for the mind to generate more ideas.
At least, that’s my feeling.
From Walmart’s personalized ‘Complete the Look’ style recommender, to why its not easy to measure the impact on conversion rate, let’s explore
This Week in Experimentation:
Blueprint of the week: Should I run an AB Test? It’s not what you get that counts, but what you have to give to get it. Link.
Talk of the week: Watch this EP of Testing Insights to see Matthias Mandiau and his team at H&M are killing it with a searchable central repository, weekly innovation meetings, and product increment objectives. Link.
Read #1: A/B Testing Guide with Google Tag Manager and GA4, written and video. Link.
Read #2: Walmart Tech: Personalized ‘Complete the Look’ model that constructs a complete outfit for the customer based on the individual items they pick on the site, with style. Link.
Read #3: UX Benchmarking method for understanding five key dimensions of user perception. Link.
Opinion of the week: “This is the real coversion rate, by day, for a client over a 12 month period. From a random point in the year, I artificially inflated all the values by 20%. Can you guess where? Link.
Event of the week: MeasureCamp North Amerika — discuss and participate in any session, everyone’s invited to speak. Free and happening on Saturday 03/11, in Remo. Link.
Blueprint of the Week: Should I Run an AB Test?
It’s not what you get that counts, but what you have to give to get it.
And today, you have to give a lot (of time and money) to get anything.
AB testing isn’t different.
It’s expensive.
Should I Run an AB Test is a framework based on a checklist from Tal Raviv.
Use Cases:
— Deciding if it's worth testing.
— Aligning and training a team to know when it's worth testing.
Talk of the Week: Testing Insights with Matthias Mandiau
H&M isn’t only about fashion.
They’re big on experimentation as well.
At the time of the recording, Matthias Mandiau was a Senior growth strategist, CRO & data analytics for H&M.
In this episode, Ben Labay asked;
1. How are you organizing your testing insights knowledge base?
2. How do you prioritize the testing ideas from your backlog?
3. What are your top struggles with the experimentation program at H&M?
Watch this episode to hear how Matthias and his team are killing it with a searchable central repository, weekly innovation meetings, and product increment objectives.
Reads of the Week:
Read #1: A/B Testing with Google Tag Manager and GA4
Google Optimize is going sunset.
So Jeroen Wierma created this A/B testing guide with Google Tag Manager and GA4.
Mike Taylor turned this into a video tutorial if you’re more into that.
The guides include everything you need:
— Prerequisites, How It Works, Variables, Triggers,
— Tags, Previewing, Scheduling, Publishing, Stopping, and Analyzing Experiments.
Read #2: Walmart Tech: Personalized ‘Complete the Look’ model
Modern eCommerces often have a plethora of choices for the customer.
The variety of choices calls for a recommendation system that guides users to find the goods they seek.
This is especially vital for style-based categories like Home and Fashion. Here, customers have unlimited choices and need recommendations from retailers to make the search fun and easy.
The objective of the ‘Complete the Look’ model is addressing such needs — developing a model generating complete, stylized outfits built around the current item the user is considering.
“To do this effectively — requires a model that not only finds the right product types (articles of clothing) that together construct a complete outfit — but also selects the right individual items that collectively complement one another from a style/brand/price/color and overall theme to create a coherent look.”
Read #3: Using UX Benchmarking to Unlock User Perceptions
If you collect data on user behavior, it's always a picture of the past. But when you try to collect data on in-the-moment user perceptions, you can’t necessarily trust the words users say. That’s because they are likely to share a rationalization rather than a true perception.
That’s why Ben Labay, Speero’s managing director, developed a new method to glean into five different dimensions of user perception.
His secret to unlocking this vault? Leveraging UX benchmarking to tap into five different dimensions of user perception.
Opinion of the Week:
— By Jonny Longden.
“This is the real coversion rate, by day, for a client over a 12 month period. From a random point in the year, I artificially inflated all the values by 20%
So, just to be clear, from a single, random day and on every day afterwards there is a 20% increase on what the real value was when I extracted the data.
Can you tell where that is?
No, you can't.
The most common question people ask after running a CRO programme for some time is: why has our conversion rate not increased?
One part of the answer is that you simply cannot 'see' increases in volatile trends. Even, if you try and do YoY comparisons, you are assuming that last year was identical to this year, which it most certainly never is.
The brain wants to imagine a nice straight line with a nice uptick in the line when things happen. Unfortunately it just isn't like this.”
Event of the Week: MeasureCamp North Amerika
MeasureCamp is an unconference, the schedule is created on the day and speakers are fellow attendees. Everyone is encouraged to discuss and participate in sessions, even to lead sessions themselves. Focus on what you find interesting and engaging.
This time, it's happening in North Amerika.
On Saturday 11 Mar 2023, in Remo.
Starting at 8AM PT / 11AM ET
This one is free and happening on Saturday, so try to visit it if you can.
Link.
And that’s it from me!
If you want me to share your podcast or blog post, just reply to this email or reach out to me personally at Martin@speero.com.
PS — Think anyone could use all this juicy info? Share the newsletter with them!
.svg)
