CRO That Ships: We Run the Tests and Write the Code

Roughly 106,000 visitors. That is what a page converting at 3% needs before it can detect a 10% relative improvement at 95% confidence and 80% power. The figure assumes the traffic is split across two variants. The number decides most of what follows. It sets which hypotheses you can afford and how many tests a quarter really holds. It also shows which reported wins were large enough to have been measured at all.

So the arithmetic comes first, out loud, before we propose a single change. How many visitors your funnel can supply, and what that sample size will and will not prove. Then we build the change as code in your repository, behind a flag. It runs to a stopping rule that was written down before the test started. That is the service. It exists for the case where you are already paying for the traffic and can see roughly where it leaks. The fix has sat behind the product roadmap for two quarters without moving up.

This service owns what happens after the click: the landing page, the funnel, the form, the signup, the checkout. It does not own getting the click. Buying and managing the traffic is media buying, and positioning and the wider demand engine sit with our SaaS growth work. Both are linked at the top of this page. We keep the line hard on purpose. The fastest way to improve a conversion rate is to narrow the traffic feeding it. A team paid for both numbers is being asked to grade its own homework.

Can your site support the test you want to run?

Ask it before you sign anything. Here is where that number above came from. It is arithmetic you can check, so you do not have to take the figure from us.

Take a page converting at 3%. To detect a 10% relative improvement, so 3% to 3.3%, at 95% confidence and 80% power, you need roughly 53,000 visitors per variant. That is about 106,000 visitors for one two-arm test. If you want to detect a 20% relative improvement instead, the requirement drops to roughly 14,000 per variant. Required sample scales with the inverse square of the effect you are looking for.

Two things follow. Small improvements stay undetectable at B2B traffic levels, no matter how long you wait, because sample requirements grow faster than patience does. And a test that ran for three weeks on 4,000 visitors and reported a 19% lift did not measure a 19% lift. It measured noise with a confident-looking label on it.

We run that calculation for every hypothesis before you commit budget, and we show you the table. Some hypotheses come back affordable. Some come back as “this needs eleven months of your current traffic”. That is a real answer and it is usually the most valuable thing in the first month.

What we do when the traffic is not there

Put your own traffic into the arithmetic above before you buy a testing programme, because the result is usually sobering. Send 40,000 visitors a month to the pages that matter and you can finish roughly one two-arm test a quarter. That is at a 10% detection threshold. A proposal will happily promise four tests a month. When the numbers come back like that, it does not mean there is no work. It means the work is different, and we will name it for what it is. Calling defect-fixing an experiment does not make it one.

• Fix the defects that do not need a test. A form that fails validation silently on iOS. A checkout step taking four seconds to render on a mid-range Android. A required field nobody can answer. You do not A/B test whether broken is worse than working.
• Test bigger differences. If you can only detect a 20% swing, propose changes capable of producing one. A new page, a different offer structure, a removed step. Not a button colour.
• Move the test up the funnel. Measure on a step with ten times the volume, where the same underlying change is detectable. Accept that you are measuring a proxy.
• Use research where statistics cannot reach. Session replay of failed sessions, and five to eight moderated usability sessions. Then read your own support tickets, which is the cheapest source of conversion insight there is and the least used.

We will say which bucket each recommendation falls into. Work that is not an experiment does not get reported with a confidence interval.

We write the code, not a document

The difference between this and a traditional engagement is the deliverable. Most conversion consultancies hand over a deck of recommendations, and the recommendations sit in a backlog behind the roadmap forever. We are an engineering team as well as a growth team, so the variant arrives as a pull request your engineers review.

That matters for the result too. Client-side testing tools inject the variant after the page loads, which delays rendering and can make the page flash through the original first. The variant is then penalised by the very mechanism that is supposed to be testing it fairly. Building variants server-side or at build time removes that bias entirely, and it is only available to a team that can commit to your repository.

Things we will not ship

A test that is winning does not get stopped early. The stopping rule was agreed in writing before traffic started, and it holds on the mornings when the graph is tempting. A lift does not get quoted without its confidence interval. A funnel we already know is underpowered does not get a testing programme so that the retainer has something to put in a slide.

Dark patterns are simply not built here, whoever asks: no fake countdowns, no invented stock counts, no cancellation flow buried three screens deep.

A seasonal upswing does not get claimed as our result either. Where we cannot attribute a change to the work, the readout says the number moved and we do not know why. That sentence costs us credit we could have taken. It is also why the log is still worth reading a year later.