‍Author: Martin P.
Title: Content Marketer

EP 56: Experimentation Decision Matrix

"The worst business decision you can make is no decision. The needs are not going to go away. Waiting is what's gotten us in the situation we're in now." — John Peace

Hey hey.

Martin P. from Speero here.

What’s up?

One question and one notification before we start.

How do you like the new format? 

I think it’s short enough to be a fun and quick read that doesn’t take too much time. 

But still, let me know what you think.

And if you didn’t know, you can send us your own blogs, talks, and frameworks for us to share in the newsletter. We will.

Now, let’s see…

This Week in Experimentation:

Blueprint of the week: Experimentation Decision Matrix: Your action plan before you start any testing. Show why the ‘winning-losing’ narrative isn’t that important.

Talk of the week: Testing Insights with Tim Mehta, Experimentation Program Manager at Microsoft Store

Speero’s CEO, Ben Labay, talked with Tim Mehta about inconclusive and flat tests, and why he no longer uses these terms when discussing experimentation. He also asked him questions like

— Do you use an explicit metrics taxonomy?

— Do you have a connection between product strategy and metric strategy?

— What guardrail program metrics are you focused on?

Read of the week: Optimizing Beyond Conversion Rate

Most website optimization agencies only look at top-of-the-funnel (ToFu) conversions. The driving questions become:

— How do we get the customer to buy?

— How do we get the customer to buy faster?

— How do we get the customer to buy more? 

While the answers to these questions may result in short-term wins, the aggressive nature of the game may mean you lose out on long-term customer retention.

There was a better way to connect businesses with customers, work with more client-side stakeholders, and ultimately make a larger business impact. 

Event of the week: UXCon Vienna

UXCon is Vienna’s first multinational event for UX Design and Research. Topics include:

— Advanced Research: advanced research methods and practical use cases.

— UX Writing: design a convo between your product and the user.

— Measuring UX: how to measure the UX of your products and use the acquired insights.

Happening September 13th & 14th, 2023, in Vienna.

Speero Story of the week: The Battle of Payment Gateways — One of Speero’s clients wanted to change their payment gateway to one with lower fees. We told them: Wait! Let’s test this first. (Read below)

Blueprint of the Week: Experimentation Decision Matrix

When you run experiments systematically, it is easy to start just testing for the sake of testing and miss the point of the actual actions and changes you want to drive through experimentation. 

That’s why Experimentation Decision Matrix Blueprint provides you with an action plan before you start any testing. 

Based on your primary success metric—and supported by the secondary ones—what's going to be your action after concluding the test? 

This way, you can remind not only you and your team, but your stakeholders as well about the actual actions you plan to take after you conclude each one of your experiments.

It will also help you enrich a narrative where “winning” and “losing” is not necessarily what’s impactful for the business.

Speero Story of the Week: The Battle of Payment Gateways

One of Speero’s clients wanted to change their payment gateway to one with lower fees.

We told them — Wait!

Let’s test this direction first to see if it's not harming conversions.

Here are the results:

Loss - Abandon

Variation resulted in:

-3.86% in Transactions (99% sig. ✓)

-2.99% in Revenue per user (using Sessions)

Est. Monthly Monetary Contribution 

(by not implementing):

$75,070 (Extra Transactions * AOV)