CRO on Shopify without guessing

Most conversion reporting is a story assembled after the numbers came in. Someone changed a button, revenue was up that week, and the change is now a win with a percentage attached. The percentage is not a measurement. It is the difference between two noisy weeks, and if you had changed nothing at all you would still have had a number.

Running conversion work so the result means something is not hard, but it requires committing to a few things before you start. Those commitments are precisely the ones people avoid, because they are what make it possible to be wrong.

Decide the one metric before you touch anything

Write down, before the change is built, the single number that decides this. Not a dashboard. Not "conversion and AOV and bounce rate". One primary metric, and the direction it has to move.

The reason for one is arithmetic. Every additional metric you are willing to declare victory on is another chance for noise to hand you a win. Watch six metrics and something will be up; the probability that at least one moves favourably by accident approaches certainty. Statisticians call this multiple comparisons. Merchants call it "well, AOV went up".

Pick the metric closest to the thing you changed, not the furthest. If you changed the product page, measure add-to-cart rate or product-page-to-checkout rate — not revenue per session, which sits downstream of shipping costs, discount codes, traffic mix and everything else that happened that week. Revenue is the thing you care about; it is a bad thing to measure, because it is the noisiest number on the site.

Secondary metrics are fine, and they have exactly one job: catching damage. If add-to-cart went up and the refund rate went up with it, you did not win, you moved a problem downstream. Declare the guardrails at the same time as the primary metric, and declare them as thresholds rather than as things you will look at.

Sample size and the stopping rule, agreed in writing

Before the test runs, two numbers go on the page: how many sessions or conversions each variant needs, and the date you will stop. Both come from your current baseline conversion rate and the smallest improvement that would actually be worth shipping.

That last input is the one people skip, and it is the one that determines whether the test is possible at all. A store converting at 2% with a couple of hundred orders a month cannot detect a 3% relative improvement. It cannot detect a 10% one either, not within a quarter. Doing the arithmetic first is often the most valuable thing that happens in a conversion project, because the honest answer is frequently "this store cannot run experiments at this traffic level" — which is not a failure, it is a redirection toward work that does not need statistics to justify it.

The stopping rule matters as much as the size. Checking a running test daily and stopping when it looks significant is a procedure that manufactures significance out of noise: a test with no real effect will cross a threshold at some point if you keep looking. Either fix the duration in advance and do not look, or use a method designed for continuous monitoring and accept its wider intervals. What you cannot do is peek and stop on the good day.

Two more rules that are cheap and prevent most of the disasters. Run for whole weeks, so weekdays and weekends are represented equally. And never start or conclude across a promotion — a test spanning a sale measures the sale.

A friction fix is not an experiment

This distinction saves more time than anything else here. There are two kinds of conversion work and they are governed by different rules.

A friction fix is a defect. The variant selector does not show which options are unavailable. Shipping cost first appears at the last step of checkout. The mobile add-to-cart button sits below three paragraphs of copy. The size guide opens a PDF. These do not need an experiment, because the counterfactual is not interesting — nobody is arguing that the broken version might be better. Ship it, note the date, move on.

An experiment is a genuine question where reasonable people disagree and the answer could go either way. Does a longer product description sell better than a shorter one? Does removing the discount code field from the cart increase completed checkouts, or annoy the people who have a code? Does a bundle raise average order value enough to pay for the units it cannibalizes? Those are worth testing, and only those are.

Most of what gets sold as a conversion programme is friction fixes dressed as experiments, because an experiment can be reported with a number and a fix cannot. That is backwards. The fixes are where most of the value sits, and pretending they are experiments burns weeks of traffic proving something everyone already knew.

Finding them is a different exercise from testing: a page-by-page read of the storefront looking for defects. That is the shape of our CRO Inspector work — inspect home, collection, product and cart, and produce a prioritized list of things that are wrong. Nothing on that list needs a test.

What you can and cannot test on the checkout

The checkout is not a page you control, and this surprises people every time. It is Shopify's, it renders from their templates, and the surface you get is defined by extension points rather than by a DOM you can rewrite. A testing tool that works by injecting JavaScript and mutating the page does not work there, and should not — a third-party script rewriting a payment form is exactly what checkout security exists to prevent.

What you can change is real, and it is code and configuration rather than a visual editor. Checkout UI extensions place your content at defined targets: a trust block near the payment section, a delivery instructions field, an offer at a specific moment. Functions change the logic itself — what a discount computes, whether a cart is allowed to proceed, which delivery options appear and in what order.

Those are testable, but the unit of variation is something you deploy, not something you toggle in a dashboard. Practically, checkout experiments are slower and coarser than storefront ones: you are comparing periods, or routing a share of traffic through your own logic, and both are more work than a visual editor. The upside is that what you learn is real. A change at that layer applies to every buyer, including the ones with script blockers who never saw your testing tool at all.

Before any of this, check that the measurement itself survives the checkout boundary. If your analytics loses attribution when a session moves from the storefront into Shopify's checkout, every conversion number you have is wrong in a way no test design fixes. Getting the tracking right is a prerequisite, not a parallel workstream.

Why most wins do not survive

Take any store with a documented history of conversion wins and add up the claimed lifts. If they were real and additive, the store would convert at a rate no store converts at. It does not. So either the wins were not real, or they did not last, and both are common enough to expect by default.

Three causes dominate. The first is stopping early, discussed above — a test halted the moment it looked good is a test that measured a fluctuation. The second is that the effect was real but small and specific to a moment: a seasonal traffic mix, a campaign, a novelty response that faded once returning visitors got used to the new layout. The third is that it never ran against a proper control at all, because the comparison was this month versus last month, and last month had different traffic.

The defence is to write the result down honestly at the time. What was tested, on what traffic, for how long, what moved and by how much, with the interval, and what the guardrail metrics did. One paragraph per test. It costs nothing, and it is the only thing that stops a programme from accumulating folklore that nobody can trace back to a number.

Then re-run the ones that mattered. If a change was worth a quarter of a point of conversion, it is worth confirming six months later that it still is. Almost nobody does this, which is precisely why the claimed lifts never add up.

What this looks like in practice

A sane conversion programme on a mid-sized Shopify store is mostly not experiments. It is: fix the defects an audit found, make the measurement trustworthy end to end, then run a small number of real questions properly — one at a time, sized honestly, stopped on schedule, written down.

That is a slower-looking programme than a dashboard full of tests, and it produces fewer numbers. The numbers it produces are true, which turns out to be the only property that compounds.