The Diagnosis:
Conversion rate is a business-wide signal shaped by pricing, brand, competitors, and market conditions, not a website KPI.
Treating it as a website problem drives unnecessary redesigns and gets experimentation programs killed for "not moving the needle."
Track program health (test velocity, decision quality, and how well tests connect to strategy) instead of staring at a single volatile trendline.
Your conversion rate has almost nothing to do with your website.
That sounds wrong. Dashboards tie it to web performance, CRO briefs promise to improve it, and redesign proposals justify their budgets by pointing at it.
I've said this flatly and repeatedly: "Improving your conversion rate is not a strategy. Yes, it's important and it could generate you a lot of revenue, but it is NOT a strategy." Put more bluntly: "The conversion rate of a website is not a purpose or vision, it's a PERCENTAGE."
But conversion rate is shaped by your pricing, your competitors, your brand, your promotions, your stock levels, your market conditions. It reflects how your customer support handles complaints and whether your logistics partner delivered on time last week. It's an indicator of literally everything about the entire business.
So why does every company treat it like a website problem? Because conversion rate is a symptom, and symptoms get blamed on whatever's closest, which is usually the website.
The concept that conversion rate equals website KPI is the root of some of the most expensive mistakes in digital: companies buy new websites they don't need, experimentation programs get killed for "not moving the needle," and CRO teams get stuck defending a number they were never in a position to own alone.
The conversion rate is a business metric that happens to be visible on the website. Once you understand that distinction, it changes how you test, how you report, and how you talk to your leadership about what experimentation actually does.
The Human Brain Wants a Simple Story
This misunderstanding is sticky because our brains are wired for linear cause and effect. Conversion rate is low. Conversion rate is measured on the website. Therefore, the website is the problem.
It's clean and intuitive, and it's dangerously wrong.
This is the exact logic behind most full website redesigns. Leadership sees a conversion rate they don't like, maps that number onto the website, and concludes the website must be broken. So they spend six or twelve months and a significant budget building a new one.
But the website was probably the least significant thing impacting that metric. In fact, targeted, research-backed fixes can outperform a full rebuild: Baymard Institute's large-scale checkout research finds the average large ecommerce site can gain roughly 35% in conversion rate through better checkout design alone, with nearly one in five shoppers abandoning a cart over a too-long or complicated checkout process.
What actually moved the number? Maybe a competitor launched a better offer. Maybe a key promotion ended. Maybe the economy shifted, and customers tightened their wallets. Maybe the product itself stopped resonating with the market. None of these show up in your site analytics. All of them show up in your conversion rate.
The conversion rate doesn't care where the problem lives. It just absorbs everything (brand perception, market conditions, operational quality, pricing strategy) and spits out a single number. Then everyone points at the website team and asks why it's not higher.
The Discount Trap: When Metrics Lie
This gets more dangerous when you realize that improving the conversion rate can actually hurt the business.
Consider a common scenario: you run heavy promotions and discounts. Conversion rate goes up. Revenue might go up too, even profit, in the short term.
But what did you just do to the customer perception of the brand? What about brand advocacy? You may have entered a race to the bottom with your competitors, one you'll never climb back out of. You attracted discount hunters instead of loyal customers. Research on the empirical relationship between discounting and brand loyalty backs this up: brands with weaker loyalty tend to rely on deeper, more frequent price promotions, a pattern that reinforces exactly the customer relationship you don't want. Your margins are thinner. Your LTV is worse.
None of these consequences show up as metrics in your A/B testing tool. They're real, and they're already doing damage.
This is the blind spot of treating conversion rate as the metric. Everything you change on your site is potentially causing wider, systemic effects that data alone can't capture. Some are positive, some are negative, and you often have no way of knowing which until it's too late.
This is why mature experimentation programs track more than just conversion rate, using frameworks like Goal Tree Maps, a Results vs Actions framework, an Experimentation Decision Matrix, a Strategic Testing Roadmap, and Program Metrics, the blueprints we use internally to separate signal from noise. A dual reporting system is one piece of this: quantitative revenue impact alongside qualitative customer insights.
The quantitative side tells you what happened to revenue. The qualitative side tells you what happened to the customer. You need both. Revenue without customer understanding is flying blind with one engine.
An A/B Test Is Not a Decision
A related trap compounds the problem. Most teams treat an A/B test result as a decision. Variant B increased conversion by 4%. Ship it.
But that test result is one piece of a puzzle, one input among many, some of which aren't data at all.
Did the winning variant align with the brand's long-term positioning? Does it create operational complexity for another team? Could it erode trust with a specific customer segment in a way that won't show up for months?
An A/B test tells you what happened in a controlled experiment over a limited window. It doesn't tell you what it means for the business. That requires judgment, context, and the kind of strategic thinking that no dashboard provides. Even mature, well-resourced programs miss more often than people expect: at Microsoft's Experimentation Platform, only about a third of tested ideas actually improved the metric they were designed to improve. That's the reality of testing, not a knock on it.
The Results vs Actions framework exists precisely for this reason. It separates the test result (what did the data say?) from the action (what did you do about it?). A test can win and still not get implemented, because the implementation would disrupt three other teams, or because the lift came from a mechanic that degrades brand perception. Separating these two lets you track your program's agility and decision quality, not just its win rate.
The Experimentation Decision Matrix takes this further. Before a test even launches, you map out what you'll do for every possible outcome. What if the primary metric wins but the secondary is flat? What if you're running a non-inferiority test where "do no harm" is the goal, not a lift? Without this plan, you fall into the trap of torturing the data until the numbers say what you want.
Improving Conversion Rate Is Not a Strategy
"Improve our conversion rate" sounds like a strategy. It isn't. It's a tactic without a direction.
A real business strategy looks more like this: shift the demographic to a younger audience, sell the first product as a loss leader and build LTV through CRM, or diversify into new audiences by creating new use cases.
Now ask: in what way does "improving your conversion rate" specifically achieve any of those? You're getting someone to buy, sure. But are they the right person? Are they buying the right thing? What does the purchase mean for the business beyond the transaction?
Experimentation can serve real business strategy, but only if you connect your testing program to strategic goals instead of pointing it at a conversion rate and hoping for the best.
Goal Tree Maps is built for exactly this. It lets you define strategic KPIs and break them down into smaller, testable metrics. Instead of "improve conversion rate," you get a tree that connects top-level business goals (35% more revenue this year) to tactical metrics (add-to-cart rate, plan selection rate) to engagement metrics (bounce rate, scroll depth). Now your tests target something specific and traceable, not a number shaped by forces outside your control.
A Strategic Testing Roadmap helps you organize those tests around themes and objectives. Rather than a backlog of random ideas, you end up with a quarterly roadmap built on "how might we" questions tied to research insights and business priorities. The whole team runs in the same direction because the direction is defined by strategy, not by a single volatile metric.
The Invisible 20%: Why You Can't "See" Your Impact
This is an illustrative thought experiment, not a real client dataset.
Imagine a client's conversion rate data plotted by day over twelve months. It's volatile: jagged peaks and valleys driven by seasonality, promotions, traffic mix, and dozens of other forces. Now imagine artificially inflating every value by 20% from a random point onward.
Can you spot where the 20% increase begins? No, you can't, and nobody can. A 20% improvement disappears inside the natural volatility of the data.
This is why the most common question after running a CRO program for a while is: "Why hasn't our conversion rate increased?" Say the team delivered real, measurable lifts across dozens of tests. When leadership looks at the overall conversion rate trendline, it looks the same, because you can't see incremental improvements in a volatile metric shaped by the entire business.
Year-over-year comparisons don't save you either. They assume last year was identical to this year. It never is. The competitive landscape is different. So is the economy, the promotions calendar, and the product mix. The comparison is fundamentally broken.
The brain wants a nice straight line with a visible uptick when good things happen. Reality doesn't work that way.
This is where Program Metrics becomes essential. Instead of staring at the overall conversion rate and wondering where your impact went, you track the health of the program itself: test velocity, quality of inputs, percentage of tests generating new insights, how many tests are backed by research. These metrics tell you whether your engine is running well, regardless of what the conversion rate trendline looks like on any given Tuesday.
Experimentation Is Not a Channel
The misunderstanding runs deep: experimentation is not a channel. It doesn't generate monthly revenue. It's not a marketing tactic.
Experimentation is a method for making better decisions about how to invest your web development spend. It's part of the overall cost of ownership and development of your digital real estate. It's a way of doing web development that tests decisions before they cost you money.
How do you measure the ROI of the total cost of an eCommerce website? Most companies compare overall cost versus overall benefit. They don't demand month-on-month attribution of specific hosting costs to specific revenue outputs. Nobody asks "what's the ROI of hosting?"
Experimentation should sit inside that same ROI framework. It's a component of how you build and improve your digital product, not a standalone revenue line.
When you treat experimentation as a channel, you set it up to fail. You demand monthly revenue attribution from a method whose value is better decisions over time. You compare it against paid media, which has a completely different economic model. And when conversion rate doesn't visibly move, you conclude the program isn't working.
Misunderstanding the framing, not bad test design, low velocity, or statistical errors, is the single biggest red herring that can doom an experimentation program.
What to Do About All This
Recognizing that conversion rate is a business metric, not a website metric, changes how you operate day to day.
- Stop promising conversion rate lifts. Promise better decisions. Promise reduced waste in web development. Promise incremental revenue impact measured at the test level, not the trendline level.
- Treat every A/B test result as one input, not a verdict. Layer in qualitative data, customer insights, brand considerations, and fit with the broader strategy before deciding to implement. Use frameworks like Results vs Actions and the Decision Matrix to make this process repeatable rather than ad hoc.
- Connect your testing program to business strategy. Use Goal Tree Maps to trace every test back to a strategic KPI. Use a Strategic Testing Roadmap to organize your work around themes, not random ideas.
- Measure your program with program metrics. Don't just stare at the overall conversion rate and hope to see a bump. Track velocity, test quality, implementation rate, and the ratio of iterative to disruptive tests. These tell you whether the program itself is healthy.
- Learn about systems thinking. Your website doesn't exist in isolation. Every change ripples outward into brand perception, customer experience, operational load, and competitive positioning. The best experimentation programs account for this. They don't just measure what happened to conversion. They ask what happened to the customer.
| Framework | What it does | When to use it |
|---|---|---|
| Goal Tree Maps | Defines strategic KPIs and breaks them down into smaller, testable metrics, connecting top-level business goals to tactical metrics (add-to-cart rate, plan selection rate) and engagement metrics (bounce rate, scroll depth). | When you need every test to trace back to a specific business goal instead of a generic "improve conversion rate" target. |
| Results vs Actions | Separates the test result (what did the data say?) from the action (what did you do about it?), so a test can win without automatically being implemented. | When implementation would disrupt other teams, or when a lift comes from a mechanic that degrades brand perception, and you need to track decision quality separately from win rate. |
| Decision Matrix | Maps out what action to take for every possible test outcome before the test launches, including non-inferiority cases where "do no harm" is the goal, not a lift. | Before any test launches, to avoid torturing the data for an answer after the fact. |
| Strategic Testing Roadmap | Organizes tests around themes and objectives into a quarterly roadmap built on "how might we" questions tied to research insights and business priorities. | When your backlog is a pile of random ideas instead of a plan the whole team is running toward. |
| Program Metrics | Tracks the health of the experimentation program itself: test velocity, quality of inputs, percentage of tests generating new insights, and how many tests are backed by research. | When you need to know if your program is working without relying on the (invisible) overall conversion rate trendline. |
The Metric Everyone Watches Is the Metric Nobody Owns
Conversion rate will always matter. It's a useful signal. But it's a signal shaped by the entire business, not by your website alone, and certainly not by your CRO team alone.
The moment you internalize this, you stop chasing a number and start building a system: one that makes better decisions, reduces waste, connects testing to strategy, and measures what it can actually control.
A program built that way is worth defending, and it doesn't need to promise conversion rate lifts to prove its value.
Frequently Asked Questions
Is conversion rate a useless metric?
No. Conversion rate is a useful signal, but it's shaped by pricing, brand, competitors, promotions, and market conditions, not just your website. The real mistake is treating it as the only metric that matters, or as something your website team can move on its own, rather than a failure to track it at all.
What should replace conversion rate as the primary KPI for an experimentation program?
Nothing replaces it outright. Pair it with program-level metrics like test velocity, the percentage of tests backed by research, implementation rate, and the ratio of iterative to disruptive tests. These tell you whether the program itself is healthy, independent of what the conversion rate trendline is doing on any given week.
Why can't leadership see the impact of a successful CRO program in the overall conversion rate?
Because a real, incremental lift, even a meaningful one, gets absorbed into the natural day-to-day volatility of a metric shaped by seasonality, promotions, traffic mix, and dozens of other forces outside the website's control. Year-over-year comparisons don't save you either, since no two years share identical conditions.
How do you connect A/B testing to business strategy instead of just conversion rate?
Trace every test back to a strategic KPI using a framework like Goal Tree Maps: connect top-level business goals to tactical metrics (add-to-cart rate, plan selection rate) and engagement metrics (bounce rate, scroll depth). Organize the roadmap around themes and research-backed "how might we" questions instead of a backlog of disconnected ideas.
About the author: Jonny Longden is Chief Growth Officer at Speero. He has spent nearly 20 years building experimentation and growth functions inside agencies, client-side teams, and management consultancies, leading product and optimization functions at Sky, Visa Europe, and Boohoo Group before joining Speero in January 2025. At R/GA, he led development of a £2 million ecommerce and optimization technology stack for Manchester United. Connect on LinkedIn.
.svg)
