A/B Testing for Conversion Rate Optimisation

A/B testing is one of the clearest ways to learn whether a proposed change caused a different outcome. Instead of showing a new page to everyone and comparing this month with last month, a valid experiment exposes comparable groups to different experiences at the same time. Random assignment helps balance the outside factors neither group controls: traffic quality, weekday mix, seasonality, promotions, and changing demand.

That does not make every A/B result trustworthy. A weak hypothesis, broken tracking, inconsistent assignment, early stopping, or the wrong success metric can produce a precise-looking answer to the wrong question. This guide explains how A/B testing should work inside conversion rate optimisation, especially for ecommerce and other multi-step customer journeys.

A/B Testing Is a Method Within CRO

CRO asks a broad question: how can the experience produce more customer and business value? It uses analytics, customer research, usability, accessibility, copy, design, technical quality, and measurement. A/B testing answers a narrower causal question: did this defined treatment change this defined outcome for this eligible audience?

A healthy relationship between the two looks like this:

1. Research identifies a meaningful problem or opportunity.
2. The team forms a specific explanation and proposed response.
3. Prioritisation determines whether the question is worth answering.
4. An experiment compares the current experience with the treatment.
5. Analysis produces a decision and an honest account of uncertainty.
6. The result updates the roadmap and the team’s understanding of customers.

A/B testing should not become a content-production machine that launches variants without evidence. Nor should teams test known defects. If a payment form is broken, a keyboard user cannot complete checkout, or tracking is wrong, fix it. Reserve experiments for working experiences where the effect of a plausible alternative is genuinely uncertain.

The Anatomy of a Valid A/B Test

Term	Meaning
Control	The current or baseline experience
Treatment or variant	The deliberately changed experience being evaluated
Eligible audience	The participants to whom the hypothesis and change apply
Randomisation unit	The entity assigned to a group, commonly a visitor, account, store, or region
Exposure	The point at which an assigned participant actually encounters the experiment
Primary metric	The single main outcome used for the decision
Secondary metric	A diagnostic outcome that helps explain the result
Guardrail	A metric that must not deteriorate beyond an acceptable level
Minimum detectable effect	The smallest difference the test is designed to detect reliably

A sound experiment needs more than two designs. It requires:

• random or otherwise justified assignment;
• stable eligibility rules;
• persistent treatment so participants do not switch experiences;
• accurate exposure and outcome tracking;
• a primary metric chosen before results are visible;
• a sample-size and stopping plan;
• comparable treatment of both groups outside the intended change; and
• a rollback path if the treatment causes harm.

Without those conditions, a dashboard may still produce a percentage and a confidence label, but the causal conclusion can be wrong.

Start With Evidence, Not a Favourite Design

Strong test ideas come from an observed behaviour or customer problem. Useful inputs include:

• funnel and cohort analysis;
• customer interviews and surveys;
• support conversations, product reviews, returns, and onsite search terms;
• session replays and heatmaps;
• usability and accessibility testing;
• technical errors and performance data; and
• sales, margin, inventory, and operational constraints.

The user-behaviour chapter explains how to combine these sources without treating one memorable replay or one customer quote as universal truth. A structured CRO audit can then turn the evidence into prioritised findings.

Write a falsifiable hypothesis

A useful hypothesis states the evidence, audience, treatment, expected outcome, mechanism, and guardrails:

Because first-time mobile visitors repeatedly open the delivery FAQ and abandon after shipping is introduced, showing the delivery range and shipping terms beside the purchase action should increase completed orders per eligible visitor by reducing uncertainty, without lowering profit per visitor or worsening page performance.

This is testable. “A cleaner page will convert better” is not. It does not define clean, explain why the change should work, identify the audience, or establish what result would matter.

Test one interpretable idea

The familiar advice to change one element at a time is useful when the team needs to attribute an effect to one element. It is not an absolute rule. If the hypothesis concerns a new product-card concept, the treatment might deliberately change the image ratio, information hierarchy, and CTA together.

The real rule is to avoid unrelated bundles. A new headline, navigation system, pricing model, and checkout field in one treatment may produce a result, but it will not reveal which idea caused it. Test a coherent concept whose result can guide a decision.

Choose Metrics That Represent the Decision

A metric hierarchy prevents a local improvement from being mistaken for a business win.

Primary metric

Choose one outcome closest to the value the treatment is meant to create. Depending on the business, that could be:

• completed orders per eligible visitor;
• net revenue or contribution profit per eligible visitor;
• qualified enquiries per eligible visitor;
• paid activation or subscription conversion;
• completed booking; or
• retained or repeat customers over a defined period.

For ecommerce, revenue per visitor is often more informative than average order value alone:

revenue per visitor = net attributed revenue / eligible assigned visitors

AOV excludes every visitor who did not order. It can rise while conversion falls enough to reduce total revenue. Profit per visitor is stronger still when product, discount, fulfilment, and return costs are reliable.

Secondary metrics

Secondary metrics explain the route to the primary outcome. Product views, CTA clicks, add-to-cart rate, checkout starts, form errors, and time to completion can show where behaviour changed. They should not overrule a negative primary outcome simply because one step improved.

Guardrails

Guardrails protect against wins that create harm elsewhere. Common examples include:

• gross or contribution profit;
• cancellations, refunds, and returns;
• lead quality and downstream close rate;
• payment or form errors;
• unsubscribes and complaints;
• page speed and layout stability;
• accessibility failures; and
• customer-support contacts.

Document exact metric definitions, attribution windows, currencies, refund rules, and data sources. If the control and treatment use different counting logic, statistical sophistication cannot repair the comparison.

Plan Sample Size and Duration Before Launch

The required sample depends on five things:

1. baseline performance;
2. the minimum effect worth detecting;
3. traffic allocation;
4. desired statistical power and error rate; and
5. the chosen frequentist, Bayesian, or sequential method.

Small effects require more observations than large effects. Rare outcomes require more traffic than frequent ones. A test powered for CTA clicks may be far too small to answer whether completed orders changed.

Use a sample-size or power calculation before building the treatment. Estimate how long it will take to collect the required eligible participants and conversions. If the expected duration is impractical, revisit the audience, outcome, minimum useful effect, or research method rather than lowering the standard after launch.

Calendar coverage still matters. Run through complete business cycles so weekday mix, campaign schedules, paydays, or sales cycles are represented. A two-week run is not automatically valid, and a four-week run is not automatically invalid; the planned sample and decision rule matter more than a generic duration.

Statistical Evidence Is Not the Whole Decision

If the test uses a fixed-horizon frequentist design, define the sample and significance threshold before launch and avoid stopping at the first p-value below the threshold. Repeated unscheduled peeking increases the chance of a false positive. A sequential or Bayesian platform can support ongoing monitoring, but only when its own decision rules are followed consistently.

At analysis, report:

• participants and conversions by group;
• absolute rates or means;
• absolute difference;
• relative lift where useful;
• confidence or credible interval;
• the primary statistical result;
• guardrail outcomes; and
• known data-quality issues or external events.

A common significance threshold such as 0.05 does not tell the team whether the effect is large enough to matter, whether the implementation cost is justified, or whether a guardrail deteriorated. Statistical significance and practical significance answer different questions.

Before looking for a winner, check experiment health. A sample-ratio mismatch—traffic materially different from the expected allocation—can signal assignment, eligibility, bot, or tracking problems. Confirm that both variants recorded exposures and outcomes, treatment persisted, events reconciled with source data, and no variant experienced disproportionate errors or inventory issues.

Ecommerce A/B Test Opportunities

These ideas are prompts to connect with evidence, not a list of changes that universally win.

Journey area	Questions an experiment could answer	Business outcomes and guardrails
Landing page and hero	Does a clearer value proposition help? Does a lightweight static image or video communicate the product more effectively?	Orders, qualified leads, or revenue per visitor; page speed and engagement as diagnostics
Navigation and discovery	Should a campaign send shoppers directly to a flagship product or a collection? Does removing low-value navigation reduce distraction or harm discovery?	Product discovery, orders, revenue per visitor, search exits
Product presentation	Lifestyle versus detail imagery; large-image, dominant-image, or thumbnail galleries; benefit-led versus specification-led copy	Orders and revenue per visitor; returns, page weight, add-to-cart as a diagnostic
Calls to action and forms	Wording, hierarchy, placement, field requirements, inline guidance, and error recovery	Completed order, booking, or qualified enquiry; errors and abandonment
Pricing and promotion	Percentage versus fixed discount; bundle framing; pricing presentation; manual versus pre-applied offer	Profit or revenue per visitor; conversion, AOV, discount cost, complaints
Cart and checkout	Shipping clarity, edit controls, payment choice, trust information, form structure, and supported urgency messages	Checkout completion, profit or revenue per visitor; payment errors, support contacts
Upsell and cross-sell	Product relevance, placement, bundle composition, and offer wording	Profit per visitor, AOV, units per order; conversion and returns
Email and lifecycle	Subject line, sender, offer, cadence, and post-purchase content	Incremental revenue, activation, retention; unsubscribes and complaints

Several nuances matter:

• Lifestyle imagery may help customers imagine ownership while detail imagery may answer fit or quality questions. Evidence should determine which uncertainty the test addresses.
• Removing navigation can reduce distraction on a focused landing page and damage exploration on a category journey. “Less is more” is a hypothesis, not a law.
• A discount can raise conversion and reduce profit. Evaluate the economics, not just completed orders.
• A countdown or scarcity message should reflect a real deadline or constraint. False urgency is a trust problem, not an optimisation tactic.
• Personalisation should be tested against a relevant default. Showing different experiences to segments does not prove that either experience is better.

A Practical A/B Testing Workflow

1. Research and prioritise

Identify the problem, affected audience, commercial value, and evidence. Rank the idea by expected impact, confidence, reach, risk, and effort.

2. Write the experiment brief

Record the hypothesis, eligibility, randomisation unit, control, treatment, primary metric, guardrails, minimum detectable effect, sample, allocation, statistical method, planned segments, start conditions, and stop conditions.

3. Build the treatment

Implement the smallest experience that validly represents the idea. Avoid adding tracking or performance differences that are unrelated to the treatment.

4. Instrument exposure and outcomes

Assignment is not the same as exposure. Record when a participant actually encounters the experiment, keep the treatment persistent, and carry the experiment identifier to the final business outcome.

5. QA both experiences

Check:

• eligibility and exclusions;
• allocation and persistence;
• mobile, desktop, browser, and assistive-technology behaviour;
• links, forms, validation, cart, checkout, and confirmation flows;
• analytics events and source-system reconciliation;
• campaign parameters and attribution;
• page speed, layout shift, and errors;
• overlapping experiments; and
• pause and rollback procedures.

6. Launch and monitor health

Monitor errors, allocation, exposure, missing events, and customer harm. Do not use early outcome movement as permission to rewrite the brief or stop opportunistically.

7. Analyse once the rule is met

Compare the primary metric and guardrails with the planned method. Treat unplanned segments and secondary metrics as exploratory unless the design controlled for them.

8. Decide and document

The result is not always “launch the winner.” A useful decision can be:

• implement the treatment;
• keep the control;
• roll back because a guardrail failed;
• run a confirmatory test;
• revise the hypothesis using new evidence; or
• label the test inconclusive because it did not achieve the planned precision.

Interpret Wins, Losses, and Inconclusive Results

A practical win

The treatment improves the primary outcome by a meaningful amount, uncertainty is acceptable, guardrails remain healthy, and implementation cost does not outweigh the value. Roll it out carefully and monitor whether the live effect remains consistent.

A useful loss

The treatment performs worse or creates harm. Keep the control, record why the original mechanism may have been wrong, and use the behavioural evidence to improve the next hypothesis. A prevented bad launch is valuable.

An inconclusive test

The interval includes both a useful gain and a meaningful loss, or the planned sample was not reached. This does not prove that A and B are equal. Decide whether the question merits a better-powered follow-up or whether resources should move elsewhere.

A surprising segment

A mobile, channel, or customer segment appears to respond differently. Check whether the segment was specified in advance, has enough observations, and makes behavioural sense. Treat an after-the-fact discovery as a new hypothesis, not an automatic personalisation rule.

Common A/B Testing Mistakes

• Using a before-and-after comparison: Promotions, seasonality, traffic, inventory, and other changes remain confounded.
• Randomising page views instead of participants: Repeat visitors can experience both variants and violate independence.
• Stopping when significance first appears: Opportunistic stopping inflates false positives in fixed-horizon tests.
• Changing the treatment mid-test: The experiment no longer evaluates one stable experience.
• Selecting the primary metric afterward: Choosing whichever metric improved turns exploration into a misleading claim.
• Testing too many variants or metrics without adjustment: Every additional comparison creates more chances for a random winner.
• Ignoring sample-ratio mismatch: An unexpected allocation can reveal a broken experiment.
• Running conflicting tests: Navigation, pricing, product-page, and checkout treatments may interact.
• Optimising a proxy: More clicks, form starts, or carts can coexist with fewer valuable outcomes.
• Ignoring new-versus-returning and device continuity: Participants may switch experiences or be counted inconsistently.
• Forgetting novelty and seasonality: A new design may attract temporary attention, while a promotion can overwhelm the treatment effect.
• Building a “Frankenstein” experience: Applying every local winner without reviewing the complete journey can leave the site inconsistent and harder to use.
• Publishing only wins: Selective memory exaggerates programme success and causes teams to repeat failed ideas.

A/B Testing and Personalisation

Segmentation describes performance for meaningful groups; personalisation assigns different experiences to those groups. Neither should begin with the assumption that one group “must prefer” a particular design.

Define the segment before analysis, ensure assignment and sample size work within it, and test the personalised experience against an appropriate default. Account for people whose segment changes, who use multiple devices, or who do not provide the data required for targeting. More variants also mean more operational complexity and more opportunities for false discoveries.

What to Do When Traffic Is Too Low

Low traffic does not justify a weak test. It changes the best research method.

• Fix technical, accessibility, and obvious usability failures directly.
• Interview customers and review support, search, return, and sales data.
• Run moderated usability tests around the highest-value tasks.
• Use surveys and session replays to sharpen the mechanism.
• Concentrate eligible traffic on one important question instead of several small tests.
• Test a larger coherent change only when evidence supports it.
• Improve the precision and reliability of the outcome metric.
• Accept “inconclusive” when the available sample cannot answer the question.

The aim is not to produce an A/B test. It is to reduce uncertainty enough to make a responsible decision.

Running A/B Tests on Shopify

Shopify adds implementation questions that a generic testing chapter should not flatten: theme and app interactions, native experiment options, persistent assignment through cart and checkout, order and refund attribution, Markets and currency behaviour, page flicker, checkout eligibility, and consistent price or shipping treatments.

Those details—including Shopify-specific test ideas such as theme, product-gallery, free-shipping threshold, pre-applied discount, and checkout configuration experiments—are covered in the complete Shopify A/B testing guide. Use that guide instead of applying generic client-side advice to a platform-aware commercial test.

Build an Experiment Library

For every test, archive:

• the evidence and hypothesis;
• audience, allocation, dates, and implementation;
• screenshots or recordings of control and treatment;
• metric definitions and analysis plan;
• raw group counts and effect estimates;
• guardrails, anomalies, and data-quality notes;
• the decision and implementation status; and
• what the result changes about future priorities.

This prevents duplicate work and protects the programme from survivorship bias. The measuring CRO success guide explains how to connect individual experiments to programme-level value rather than celebrating a stream of isolated dashboard wins.

The Point of A/B Testing

A/B testing does not replace judgement. It disciplines judgement. It forces a team to state what it believes, expose comparable audiences to a deliberate difference, measure an outcome consistently, and confront uncertainty.

Used inside a research-led CRO programme, an experiment can validate a valuable change, prevent a harmful launch, or reveal that the team misunderstood the customer problem. All three outcomes are progress when the test is designed to be trusted.