What to Test When You Don't Have the Traffic

Most CRO advice assumes you have the traffic to run clean A/B tests, and most stores don't. Trying to run statistical tests on low traffic produces confident-looking results that are really just noise, which is worse than not testing at all. When you don't have the traffic, the answer isn't to test anyway and hope, or to give up on evidence, it's to switch from statistical testing to other forms of evidence, and save the few real tests you can run for the biggest changes on your highest-traffic pages.

This is a core of the Experimentation & CRO Methodology pillar, under the cornerstone that most of what you believe about your store is a guess. The cornerstone said CRO is a way of thinking, holding beliefs as hypotheses and seeking evidence. This piece is about what to do when the evidence you can't afford is a big A/B test.

The dirty secret of A/B testing advice

Almost all the testing advice you'll read assumes high traffic. Run the test, wait for statistical significance, ship the winner. It sounds clean and scientific. And it quietly assumes a volume of visitors most stores simply don't have, because statistical significance needs a lot of traffic, more than most pages on most stores will ever see in a reasonable testing window.

So smaller stores land in one of two bad places. They don't test at all, and stay running on the accumulated guesses the cornerstone warned about. Or, worse, they run A/B tests that never actually reach significance, and read the noise as a result. Both are avoidable once you stop equating "evidence" with "a big A/B test."

Why low-traffic A/B testing is worse than none

This is the trap that catches the well-intentioned. With too little traffic, your test result is dominated by random variation. You watch the dashboard, see "version B is up 12%," and ship it, except that difference is noise. Run the same test again and B might "lose" by 8%. You didn't measure a real effect; you measured the random wobble of small numbers.

Acting on underpowered tests means confidently shipping essentially random changes while believing you're being rigorously data-driven. That false confidence is genuinely worse than openly guessing, because the guesser at least knows they're guessing and stays appropriately humble. The underpowered tester has a number, a chart, and a completely unjustified certainty. Even with a solid hypothesis, an underpowered test can't validate it, it just can't see the signal through the noise.

Low traffic means different evidence, not no evidence

Here's the reframe. The mistake is treating "evidence" as a synonym for "A/B test." There's a whole class of evidence that needs no statistical significance at all, because it's qualitative: it tells you why, not how many. And finding out why people struggle doesn't require a huge sample:

• Session recordings and heatmaps. Watch where real people hesitate, rage-click, scroll past the thing, and abandon. You don't need a thousand sessions to see a pattern; you need to actually watch a few.
• User testing. Watch a handful of real people try to complete a task on your store. A long-standing finding in usability research is that even a small number of users surfaces most of the serious usability problems, you don't need statistical power to see someone fail to find the checkout.
• Surveys and just asking. A short prompt ("what nearly stopped you buying today?") or a few customer conversations will tell you things no conversion number can.
• Funnel analysis in your own analytics. Where do people drop off? That's not a test you have to run, it's already happening in your data, waiting to be read.
• Support and service patterns. The same confusion reported again and again by customers is evidence. If ten people asked where the size guide is, the size guide is too hard to find.

None of these prove a conversion lift the way a powered A/B test would. But they reliably find problems, and fixing an obvious problem you've watched five real people hit does not need an A/B test to justify. You already have the evidence; it just isn't a p-value.

How to spend the few real tests you have

When you can run a quantitative test, spend it like the scarce resource it is:

• Test only where the traffic is. Your checkout, your top few product pages, your homepage, the high-traffic surfaces have a chance of reaching significance. Running a test on a low-traffic page that will never get there just wastes weeks and produces noise.
• Make bigger swings. This is counterintuitive but important: a tiny change (a button colour) produces a tiny effect that needs an enormous sample to detect. A bold change (a genuinely redesigned checkout flow) produces a bigger effect that shows up in far less traffic. Low traffic actively favours big bets over micro-tweaks, the opposite of the "test everything, one pixel at a time" advice.
• Accept longer run times, and don't peek. Low traffic means a real test takes longer, often weeks. Let it run its planned course rather than stopping the moment it looks good, which is its own serious mistake.

Fewer, bigger, properly-run tests beat a flurry of underpowered ones every time.

Judgment is still allowed

Some decisions you'll make on qualitative evidence plus judgment, without a clean statistical test, and that is legitimate. Watching five users fail to find your returns policy is more evidence than a senior person's hunch, even without a p-value attached. The cornerstone's point was that CRO is a way of thinking, holding your beliefs as hypotheses and respecting evidence over opinion. Qualitative evidence is still evidence.

The one thing to guard against is letting "we can't A/B test it" quietly become an excuse to slide back into pure opinion-based decisions. Not being able to run a powered test doesn't license going back to the HiPPO. It means reaching for the other evidence, watching, asking, reading the funnel, instead of throwing up your hands.

What this comes down to

Low traffic doesn't mean no evidence; it means different evidence. Trying to run statistical A/B tests you don't have the volume for just dresses up noise as insight, which is worse than honestly admitting you're guessing. Use the evidence you can actually get, watch real people use your store, read your funnel, ask your customers, to find the obvious problems, and save your scarce real tests for big swings on your highest-traffic pages.

The cult of statistical significance has convinced a lot of small stores they can't do CRO at all, which is exactly backwards: they often have the most to gain and the clearest problems, they just need to use the evidence available to them instead of the evidence they can't afford. The goal was never to run tests. It was to stop being wrong, and you can do a great deal of that without ever reaching significance.

A few common questions

Can I do CRO if I don't have much traffic? Yes, just not primarily through A/B testing. Statistical significance needs a lot of traffic, more than most pages on most stores will see in a reasonable window, so running A/B tests on low traffic tends to produce noise dressed up as results. Instead, lean on qualitative evidence that needs no significance: session recordings, watching a handful of real users, surveys, funnel analysis, and support patterns. These reliably find problems even with small numbers. Smaller stores often have the most to gain from CRO; they just need to use the evidence they can afford.

Why is running A/B tests on low traffic worse than not testing? Because the result is dominated by random variation. You see "version B is up 12%," ship it, and the difference was noise, run it again and B might lose. Acting on underpowered tests means confidently shipping essentially random changes while believing you're being data-driven, and that false confidence is worse than openly guessing, because a guesser at least stays humble. The underpowered tester has a chart and an unjustified certainty. Even a solid hypothesis can't be validated by a test that can't see the signal through the noise.

What evidence can I use instead of A/B tests? Qualitative evidence, which tells you why rather than how many: session recordings and heatmaps (watch where people hesitate and abandon), user testing (a handful of users surfaces most serious usability problems), surveys and customer conversations ("what nearly stopped you buying?"), funnel analysis in your own analytics (where people drop off, no test needed), and support patterns (repeated confusion is evidence). None prove a conversion lift, but they reliably find problems, and fixing an obvious problem you've watched real people hit doesn't need an A/B test to justify.

When should I actually run an A/B test on a smaller store? Save your scarce tests for the highest-traffic pages (checkout, top product pages, homepage) where they can reach significance, and make bigger swings rather than micro-tweaks, since a bold change produces a larger effect that shows up in less traffic, while a tiny change needs an enormous sample. Accept that real tests take longer on low traffic and let them run their planned course without peeking. Fewer, bigger, properly-run tests beat a flurry of underpowered ones, and many decisions can be made well on qualitative evidence plus judgment.