‍Author: Martin P.
Title: Content Marketer

EP 54: Sample Ratio Mismatch Alert

We're not just limited by technology but by our ideas and our experimentation and how quickly we can try things. — Tim Sweeney

Yo.

Martin P. from Speero here.

Here’s what has been happening..

This Week in Experimentation:

Blueprint of the week: SRM Alert — You have set the ratio for your test at 50/50 and your testing tool has reported that you have a 70/30 traffic split in your experiment. What to do?

Talk of the week: Learn how many A/B tests you can run with Bandwidth calculations for Experimentation programs [GA4]

Learn: 

— How many A/B tests can I run per year?

— What is the uplift (MDE) I can measure with my test

— How long should my experiment run

— How to find the data in GA4 and which tools to use

Read of the week: In Uncertain Times, Embrace Imperfectionism

Everyone knows about Amazon Prime or Zappos. It’s less likely you spotted the baby steps Amazon took to expand into consumer financial services.

Amazon acquired TextPayMe, invested in Bill Me Later, and launched a remote card payment device, Amazon Local Register. All modest, less-known moves. 

Most ended in apparent failure — TextPayMe closed in 2014, Local Register withdrew from the market, and Bill Me Later was acquired by Paypal.

Yet today Amazon is a powerhouse in consumer finance, boasting a 24% user share in the U.S. for its pan-economy Amazon Pay service. How did the company craft success at scale from a series of pint-sized and seemingly unpromising moves?

The answer is that Amazon is an imperfectionist. 

Imperfectionism is an approach in which companies grow not by following a strategic framework or plan, but through multiple and frequent experimentation in real time, incrementally building up valuable knowledge, assets, and capabilities on the way. Link.

Event of the week: CXL Live 2023 (in Speero + CXL organization) — the top experimentation and growth event that’s about relationships, not PowerPoint. 

This event focuses on building authentic relationships and spending quality time with other like-minded people. You are guaranteed to meet people with a similar role at a similar company working on the same challenges that you are.

That’s not to say content doesn’t matter: it does, and we’ve always brought our A-game. But high-quality content is table stakes. And in the end, other things matter more.

Speero Story of the week: Full Page Radical Redesign — We had a long list of recommendations and tweaks for a client. Speero made a big bet and decided to combine them all in a single test. The test won, leading to more transactions, adds to cart, revenue, and AOV.

Blueprint of the Week: SRM Alert

You have set the ratio for your test at 50/50 and your testing tool has reported that you have a 70/30 traffic split in your experiment. What to do?

There are several ways that your experiment can have a sample size mismatch, but the most common will be technical issues with the segmentation itself.

Use this framework as a guideline to determine what is the best next step once you have received a sample ratio mismatch error.

Speero Story of the Week: Full Page Radical Redesign

We had a long list of recommendations and tweaks for a client. 

Speero made a big bet and decided to combine them all in a single test. 

We introduced a new Product Landing Page design including multiple improvements (some of which were tested with success before), hypothesizing that the value perception of the brand will increase and more transactions will be completed. 

The result?

+5.47% in Transactions (97% sig.)

+2.28% in Adds to Cart (91% sig.)

+9.70% in Revenue per user (using Sessions)

+4.00% in AOV

Est. Monthly Monetary Contribution:

$35,916 (Extra Transactions * AOV)