‍Author: Martin P.
Title: Content Marketer

EP 58: Validation Methods

"Testing leads to failure, and failure leads to understanding." — Burt Rutan

Hey hey.

Your host is Martin P.

Glad to see you here.

As always, we’ll present the best posts, podcasts, blueprints, and events in the world of experimentation.

This Week in Experimentation:

Blueprint of the week: Validation Methods — Explore all validation methods you can use to test site changes when you simply don’t have enough traffic for A/B testing.

Talk of the week: Make Personalization Personal with Care and Competence with David Mannheim.

David is here to talk about why personalization is vital in marketing. But for him, personalization is more a communication principle than just a marketing gimmick. He also talked about:

— Challenges you will face in implementing personalization.

— The pressure for instant results vs balancing the metrics.

— The need for brands to concentrate on developing relationships with their customers based on care and trust.

Read of the week: The Inequality of A/B Testing.

Microsoft has published a paper about the distribution of their A/B test results. And the paper has enormous implications for most experimenters. 

The main insight? Optimal experimentation is based on whether you get most of your results from typical innovations OR from unpredictable and rare big successes you can detect with small sample tests.

But when this distribution is very fat-tailed, a better approach is trying out more ideas, each with possibly smaller sample sizes. Key takeaways:

— If your results are ‘marginally’ or ‘barely significant’, assume they aren’t significant at all.

— If you need an A/B test to tell you if something is better or worse, it probably doesn’t matter.

— General conversion rate improvements come from a couple of big wins, not from dozens of incremental improvements.

Event of the week: CXL Live 2023 (in Speero + CXL organization) — Where all the top people in experimentation, growth, and demand generation come together. 

To listen to keynotes? No. 

Well, that as well. 

But CXL Live is about something more important.

CXL Live is about making lasting relationships.

You meet with people who share the roles, goals, problems, and KPIs you do.

You discuss and solve them together in mini-groups.

You meet each other.

You party together. 

You leave the conference with something more than 1-2 listens of content you can see on YouTube.

October 16-17. Austin, TX. 

Speero Story of the week: Show Them the Free Shipping — Here’s how Speero tested letting them ‘buy now and pay later’, and increased transactions, AOV, and revenue per session. (Read below)

Blueprint of the Week: Validation Methods

So, you’re making changes to the website. 

But shipping out big changes without validating them first can be quite expensive.

It can even create more issues than it solves. 

You need to test disruptive ideas before releasing them into the wild.

Speero’s Validation Methods Blueprint helps with just that.

It helps you decide which validation method you should use when you’re making site changes. Even if you don’t have enough traffic for A/B tests.

For instance, using user testing for validating functionality changes instead of A/B testing. 

Or using copy testing tools like Wyner to validate copy changes when you don’t have enough traffic for A/B testing.

There’s also an in-depth video explanation of the blueprint, take a look.

Speero Story of the Week: Buy Now Pay Later.

Here’s how Speero tested letting them ‘buy now and pay later’, and increased transactions, AOV, and revenue per session.

We wanted to find out whether adding a “Buy Now Pay Later” solution will have a positive or negative impact on the client’s site performance.

Basically, if we enable AfterPay on the website, more people will be able to buy the products right away, leading to more AOV and transactions.

The results were conclusive.

+2.97% in Transactions (99% CTBC)

+4.55% in Rev. per session (99% CTBC)

+1.63% in AOV

Est. Monthly Monetary Contribution*

$88,655 (Extra Transactions * AOV)

Our recommendation? Implement!