How to test high-ticket stores when buyers take weeks to decide

A shopper can leave your site four times and still be moving toward a purchase. If they are choosing a mattress, sofa or e-bike, those visits may be part of the decision: compare the options, ask someone else, check delivery, then come back.

That changes what we should research and how we should test. A higher session conversion rate can reflect fewer visits without more buyers. An early rise in add-to-cart can disappear by checkout. A test can finish before its newest participants have had time to decide.

Speero (back when we were CXL Agency) ran research and testing programs for a run of bed-in-a-box brands, including Nectar, Tuft & Needle, Saatva and Tomorrow Sleep. Most of what we learned came from polls, interviews, usability sessions and analytics. That kind of research tells you where to test. It doesn’t tell you what will win, so below we say which findings reached a concluded test and which didn’t.

This guide is for ecommerce teams whose customers take time to decide. Price is a clue, not a diagnosis. A first-time supplement buyer may research for weeks, and someone replacing a broken fridge may buy today. What matters is whether your shopper is restocking or making a decision.

Ecommerce is two businesses, not one

“Ecommerce” lumps together two different kinds of store, and in our view most CRO advice treats them as one. Melvin Copeland drew the line more than a century ago, sorting goods into convenience, shopping and specialty goods. Convenience goods are bought often and with little effort. Shopping goods are bought after comparing price, quality and fit across options. It’s still the most useful split we know for deciding how to optimize a store.

We’ve tested both kinds for years. On convenience products like deodorant and flowers, a large share of visits end in a purchase, because the buyer already knows what they want and the site’s job is to get out of the way. On shopping products like mattresses and sofas, the median home and furniture store in Littledata’s Shopify benchmark converts about one session in a hundred.

A gap that size reflects a different buying process, and the playbook has to change with it. Our read is that much of the published CRO advice comes from high-volume convenience stores, where tests reach significance fast. Urgency timers, discount pop-ups and shorter checkouts suit a buyer who has already decided. A mattress shopper on their third visit is still deciding.

The rest of this guide is about that second kind of store.

Look at the journey behind your numbers

In our Tuft & Needle research, buyers described trying a friend’s bed and researching elsewhere before arriving at the site. A purchase on the first recorded day does not necessarily mean a short decision.

First recorded visit, first exposure to the brand and start of consideration are different events. Analytics helps with the first. Research is needed to understand what happened before it and between visits.

Choose a metric that survives repeat visits

Session conversion remains useful for describing traffic. It is a fragile primary scoreboard when an intervention changes how often someone returns. The denominator can move even when buyers and revenue do not. We make the longer case in optimizing beyond conversion rate.

For an experiment spanning visits, define the outcome per unit assigned to the test. If assignment uses a persistent browser ID, call it a browser ID. Do not describe it as a known person across devices. Aggregate outcomes and calculate uncertainty consistently with that assignment unit.

A practical revenue measure is revenue per assigned eligible visitor over a fixed follow-up window. Include everyone assigned under the eligibility rules, including people who buy nothing or ignore the new feature. Specify whether revenue includes tax, shipping, discounts, cancellations and refunds.

Use buyer conversion as a supporting outcome. Average order value helps explain the result, but it is conditional on buying: a change in who purchases can change AOV.

For financing, discount or return-policy tests, consider contribution per assigned visitor as the primary outcome when reliable cost data is available. Otherwise, pair revenue with guardrails for relevant costs and customer outcomes. More sales can come with higher payment fees, fulfillment costs or returns.

Do not automatically cap the largest orders

Large legitimate orders may be exactly what an intervention changes. Investigate duplicate events and tracking errors separately. If you use capping or winsorization to reduce noise, predefine the rule, its threshold and whether it applies per order or per visitor. Report a raw-revenue sensitivity check and explain any disagreement.

A capped metric estimates a different quantity from total revenue. Optimizely’s documentation also distinguishes event-level from user-level adjustment; these can produce different results. Check the method your specific platform uses. Alex Birkett’s guide to outliers on CXL covers the options, and our post on what most A/B test calculators get wrong explains why revenue metrics need a variance input when you size the test.

Verify assignment across return visits

Test what happens after a gap, on login and across the browsers customers use. A shopper who receives a new identifier may be reassigned. A browser breakdown can reveal symptoms; it does not repair them.

Server-set cookies alone do not guarantee continuity. WebKit restricts script-writeable storage and some server-set cookies under its tracking-prevention rules. Check the current rules against your implementation. CXL’s guide to attribution models describes the same problem from the analytics side: long journeys split into fragments that look like new visitors.

Where customers identify themselves, use permitted deterministic links and predefine how assignment conflicts are handled. Measure the share of journeys you can join. Do not assume the anonymous remainder is complete, or exclude only treatment users whose identities are difficult to resolve.

How long should a long-consideration test run?

Separate enrollment from follow-up. Enrollment recruits the planned sample. Follow-up gives each participant time to produce the outcome.

Suppose enrollment takes four weeks and every participant gets 28 days after assignment to purchase. The last cohort matures around week eight. Reading the result at week four would give late entrants much less opportunity to buy.

Choose the follow-up window using sufficiently mature historical cohorts and the business decision. Inspect the distribution of purchase delays and the cumulative share purchasing by each horizon. A mean among recent buyers misses people who have not bought yet. First recorded visit also misses earlier research. A cohort analysis by first visit is a practical way to see the delay curve.

Predefine eligibility, sample requirements, minimum calendar coverage, maximum enrollment period, outcome window and stopping method. Account for promotions, stock constraints and other calendar changes. Keep each participant’s assigned experience available throughout follow-up, even after new enrollment stops.

Monitor errors and harms while the test runs. For conventional fixed-horizon inference, avoid stopping because the result first becomes significant. If using sequential inference, follow the method’s rules. Sequential testing addresses repeated looks; it does not make delayed purchases arrive sooner. Peep Laja’s guide to stopping A/B tests and CXL’s piece on peeking and sequential testing go deeper.

Choose a separate later readout for post-delivery outcomes. A 28-day revenue result cannot establish the full economics of a year-long home trial. Report the maturity of cancellation and return data alongside the result.

If available traffic cannot distinguish an economically useful effect, resolve that before launch. Larger changes may have larger effects, but they still need a feasibility check, as in our low-traffic case study. An inconclusive experiment is not evidence that the change has no effect.

Six uncertainties to turn into experiments

The recurring themes in our mattress work are useful starting points. Treat each as a question to investigate in your market. Use ResearchXL to connect qualitative observations with analytics and technical checks before prioritizing an intervention.

1. Which option fits me?

In Nectar usability work, some participants did not discover the comparison page; those who did found it useful. Visitors who viewed comparison content also converted at a higher rate in the analytics. Motivated shoppers compare more, so that’s a correlation, but we read it as a sign that comparison is where these shoppers decide.

Investigate whether shoppers can explain meaningful differences between models. On desktop, a side-by-side view may help. On mobile, start with consistent attribute order, concise differences summaries and a saved shortlist.

Baymard’s comparison research found much less use of comparison tools in its mobile sample than on spec-driven desktop sites. So test how comparison works on mobile rather than dropping it. Peep Laja made the short version of this point years ago: people comparison shop, and the more expensive the product, the longer they take. CXL’s product comparison guidelines are a useful build checklist.

Test next: a differences summary against current product content. Check comprehension before launch, then evaluate the declared business outcome. More comparison clicks could mean progress or confusion.

2. What happens if I choose wrong?

Our Nectar and Tomorrow Sleep research found that participants missed trial or return information. Offering reassurance is not the same as making it findable and understandable. We cover the wider problem in closing the sensory gap in online shopping.

A Nectar mobile checkout experiment combined earlier benefits, easier add-ons and urgency. The result leaned positive but wasn’t statistically conclusive, and because it bundled several changes it doesn’t isolate the trial message.

Test next: change the position and explanation of existing trial terms while keeping the policy unchanged. Ask participants to explain eligibility, return costs and the process in their own words. Evaluate purchases alongside cancellations and later returns. Test policy generosity separately if that is the business question.

3. Can I judge fit without trying it?

At Tomorrow Sleep, poll responses asked how to choose between firmness options. At Saatva, shoppers sought details such as sleeping temperature, motion transfer and dimensions. “I cannot try it” can hide several different information gaps.

A product finder is one possible response. Clearer specifications, demonstrations or access to a person may solve a different part of the problem. Every mattress research report we wrote recommended a finder, and none reached a concluded test in our archive. We’ve written more on ecommerce quizzes.

Test next: guidance that explains why a model fits a stated need and lets shoppers explore alternatives. Check recommendation accuracy against product constraints. Follow purchases through to fit-related returns. Treat quiz completions as a leading indicator, not the result.

4. Is the price wrong, or the timing?

Tuft & Needle research included shoppers looking for a missed sale or a discount code. Tomorrow Sleep feedback also raised the cash tied up during a trial. These are different objections from believing the product offers poor value.

Research the objection before choosing a treatment. Financing may help with cash timing, but it will not explain an unclear value proposition. Present the full price and repayment terms clearly; credit-impact and eligibility wording must match the actual provider and product. CXL’s product page guidelines recommend showing payment options on the product page for expensive items. For testing prices themselves, see our price elasticity primer.

Test next: a clearer payment explanation when the research supports it. Measure contribution after relevant fees, purchases and cancellations. Preserve a later readout where purchase timing matters: a promotion can pull orders forward. Use genuine deadlines; shoppers returning over several weeks can notice a resetting countdown.

5. Will it arrive when I need it?

Delivery questions appeared in Nectar chat logs, while Tuft & Needle research identified frustration with uncertain arrival timing. The useful question is whether the delivery promise lets someone plan their purchase.

Test next: an arrival estimate near the product decision, using reliable operational data. Distinguish dispatch from arrival and explain postcode or availability dependencies. Use an exact date when dependable; otherwise give an honest window. Track orders, delivery-query contacts, cancellations and whether the promise was met.

6. Can I trust the evidence?

Saatva post-purchase research highlighted reviews and recommendations, while scroll analysis suggested that many visitors did not reach reviews on the main product page. Buyers said reviews decided it, and most visitors never reached them. That gap is worth a test.

Test next: specific owner evidence near the product claims it addresses, with access to the wider review set. Let shoppers find experiences relevant to their needs. Check whether moving reviews displaces important fit information. Judge it on sales, not on how many people read reviews.

Research what happens between visits

Start with actual decisions. Recruit recent buyers, people still considering and people who left without purchasing. Include post-delivery interviews with people who kept or returned the product. Post-purchase surveys and support logs alone miss people who never bought or contacted you.

Sample for meaningful differences: first-time versus replacement purchases, urgent versus discretionary needs, and solo versus shared decisions. Record who participated and how they were recruited. Theme frequency in a small purposive sample is not a population estimate.

Use short diary entries during consideration to capture new questions, competitor visits and conversations while they are fresh. Follow up with interviews and, where participants agree, examples of material they used. Retrospective interviews add context, but memory can simplify a messy journey.

Useful prompts include:

• What happened that made you start looking?
• What did you do between visits?
• Which alternative did you seriously consider, and why?
• What information changed your decision?
• Who else was involved, and what did they need to know?
• What did you discover after delivery that you could not judge beforehand?

In mobile usability sessions, ask participants to choose an option that meets their needs and explain the trade-offs. Do not direct them to the quiz or comparison feature. Observe whether they can find differences, retain a shortlist and share it without losing context.

For each theme, preserve the source, participant context, observation, interpretation, competing explanation and next study. A stated preference is a hypothesis to test, not a result.

A worked experiment brief

Illustrative scenario: shoppers struggle to distinguish two sofa models. The proposed experiment compares a concise differences summary with the current product content. This is an example plan, not a client result.

Question. Does clearer fit information improve purchase outcomes within the declared follow-up window?

Eligibility and assignment. Assign eligible product-page visitors before either version renders. Persist assignment using the documented browser or account key. Analyze everyone assigned, including those who ignore the summary.

Primary outcome. Revenue per assigned eligible visitor during a fixed 28-day follow-up, if historical data supports that horizon. Choose the revenue definition before launch.

Diagnostics and guardrails. Use comprehension work and summary interactions to understand behavior. Monitor contribution, cancellation, page performance and fit-related returns on their appropriate timelines.

Feasibility. Calculate sample requirements using eligible traffic, baseline visitor-level revenue variance, the minimum useful effect, power and the chosen method. Fit enrollment and follow-up to the business calendar before committing.

Decision. Compare the estimate and uncertainty interval with pre-agreed decision thresholds. Ship a supported useful improvement with acceptable guardrails; reject material harm. An interval spanning useful gain and material loss leaves the question unresolved. A rollout under uncertainty is a business choice, not a demonstrated win.

Run it like a B2B funnel

Our Tomorrow Sleep research described a site built for a buyer who was ready to purchase, while most visitors weren’t ready yet. The recommendation was the kind a B2B marketer would make: educate, capture an email, and give people a guided way to build their choice. Each visit should move the shopper a step closer and give you a way to stay in touch.

The comparison holds up. Gartner describes B2B buying as a set of jobs: identify the problem, explore solutions, build requirements, pick a supplier, validate and reach consensus (Gartner). A couple buying a high-end bed does a small version of each, often with one person researching and the other signing off. McKinsey’s consumer decision journey research found that the set of brands under consideration can grow during evaluation, and that most touchpoints in that phase are driven by the shopper, like reviews and word of mouth (McKinsey, 2009).

Long consideration benefits from continuity: a saved configuration, a shareable shortlist, useful follow-up or help from a person. These needs can occur on any visit. Avoid treating a first visit, return visit and buying visit as observable, fixed stages.

Maintain one backlog across site, CRM, service, merchandising, finance and operations. Service supplies recurring questions; merchandising and operations validate product and delivery promises; CRM supports the gap between visits. Whether that backlog is run by a central team or a center of excellence depends on the company; CXL’s guide to structuring optimization teams covers the trade-offs.

If an experiment changes email capture and follow-up together, define the intervention as the combined experience. Use a randomized comparison among eligible shoppers and measure outcomes by original assignment. Comparing subscribers with non-subscribers, or counting attributed email revenue, does not establish incremental value.

A goal tree can connect these activities to business objectives, so quiz completions, saved shortlists and email signups are tracked as leading indicators while tests are judged on revenue. We cover how to pick those layers in experimentation metrics that matter.

Before your next test

1. Identify the uncertainty. What evidence says it matters to this customer group?
2. Define the unit. Who is eligible, how are they assigned, and will assignment persist?
3. Define the outcome. What counts, who stays in the denominator, and when does it mature?
4. Check feasibility. Can available traffic detect a useful effect within the business calendar?
5. Protect the economics. Which costs, returns or customer outcomes could make a sales increase unattractive?
6. Agree the decision. What happens after a positive, harmful or inconclusive result?

If you sell something people think about for weeks, your site is part of a sales process. Research the gaps between visits, keep the journey connected, and judge each test on outcomes that have had time to arrive.

Research findings in this article come from Speero client programs and are described directionally. The two convenience-brand rates in the first figure are rounded, and the brands are not named.