Why Most A/B Tests Are Inconclusive (and Why That's Not the Problem)

Most A/B tests come back inconclusive, with no clear winner, and that's not a failure of the test, it's the honest and common result of testing changes that mostly have small effects. The two ways people get this wrong are peeking and stopping the moment a result looks good (which manufactures false positives) and reading a flat result as "it works a little." The patience to let a test finish and the honesty to call a flat result flat is most of what separates real testing from theatre.

This is a core of the Experimentation & CRO Methodology pillar, under the cornerstone that most of what you believe about your store is a guess. The cornerstone said most tests come back flat and that's a sign of a healthy programme. This piece is about resisting the two temptations that make people fake a result instead.

Inconclusive is the normal result

People expect a test to produce a winner. Mostly, it produces "no significant difference." That feels like the test failed, like you wasted weeks for nothing. It didn't fail. It told you the truth: this change didn't move things meaningfully.

And the cornerstone's point explains why that's so common, most changes genuinely have small or no effect on behaviour. So most tests of those changes come back inconclusive. That's the system working exactly as intended, not breaking. An inconclusive result is a real answer to a real question, the answer just happens to be "this doesn't matter much," which is useful to know. The trouble starts when people refuse to accept that answer and go looking for a winner that isn't there.

The peeking problem

This is the single most important discipline point in testing, and the most commonly violated. People watch the test live, and the moment it crosses the line into statistical significance, they stop it and declare a winner. It feels efficient, why wait once you've got your result?

The problem: a test result fluctuates constantly as data comes in. Early on, with small numbers, it swings around a lot, and at some point it will very likely cross the significance threshold by pure chance, then cross back. If you stop the instant it looks significant, you're cherry-picking a random high point, and you'll declare "winners" that are pure noise far more often than the statistics suggest. Peeking and stopping early dramatically inflates your false-positive rate. You'll "find" wins that evaporate the moment you ship them.

The fix is simple and non-negotiable: decide the sample size or duration in advance, and don't stop early. Significance is only meaningful at the planned endpoint, not at the first moment it flashes green mid-test. The discipline to ignore the tempting early result is what makes the final one trustworthy.

The "it works a little" problem

The second self-deception runs the other way. The test comes back flat, not significant, but you wanted your idea to win, so you squint at it and decide "well, version B was slightly ahead, let's just ship it." This feels reasonable. It isn't.

"Not significant" means you cannot distinguish the result from no effect at all. "Slightly ahead but not significant" is not "a small win", it is "we genuinely don't know," because that slight lead is exactly the kind of difference random noise produces. Shipping version B because it was marginally ahead in a flat test is shipping on hope, with a chart attached to make it feel like data. The honest call is the boring one: inconclusive means inconclusive.

Why faking it defeats the whole point

Both sins, peeking and over-reading, destroy the entire value of testing. The cornerstone said testing's worth is that it stops you being confidently wrong. But if you stop tests early at random highs and read flat results as small wins, you've turned a truth-finding instrument into a rubber stamp for whatever you already wanted to do. That's worse than not testing, because now your opinions come pre-laundered as evidence, and everyone trusts them more, not less. A testing programme without the discipline isn't science, it's theatre with a dashboard.

What to actually do with an inconclusive result

An inconclusive test is genuinely useful, once you stop wishing it were a winner:

• It tells you this change doesn't matter much, so stop investing effort in it and move on to a bigger lever.
• It frees you to choose on other grounds. If both versions perform the same, ship whichever is simpler, cheaper, or easier to maintain, the test just told you the choice doesn't cost you conversions either way.
• It sharpens your next hypothesis. Your reasoning about why this change would win was apparently wrong; that's information about your customers you didn't have before. Update and aim better next time.

A flat result handled honestly moves you forward. A flat result tortured into a fake winner sends you confidently backward.

Patience is a CRO skill

All of this comes down to patience, which is an underrated testing skill. Tests need to run their planned course, and on lower traffic that means weeks, during which the result will tempt you to stop early at every random swing. The temptation to call it early is strongest exactly when, on thin data, you can least afford the false positive. Sitting on your hands and letting the test finish is doing the work, even though it feels like doing nothing.

The same patience argues for fewer, bigger, properly-run tests over a flurry of tiny ones. A hundred underpowered, peeked-at tests produce a hundred confident wrong answers. A handful of patient, well-powered ones produce a few trustworthy ones, which is the entire point.

What this comes down to

Most A/B tests are inconclusive because most changes don't do much, and the honest response is to accept that, not to manufacture a winner by stopping early or reading hope into noise. The patience to let a test finish and the discipline to call a flat result flat are not the boring parts of testing you get through to reach the fun part, they are the testing. Everything else is just running variations.

Be suspicious of the dramatic case studies too, the "we boosted conversions 47% with one button change" stories are mostly survivorship bias and bad statistics: the flat tests never get written up, and plenty of the headline wins wouldn't survive being re-run. Real testing is mostly undramatic. A programme that's honest about its flat results is doing real science. One that always finds a winner is usually just very good at finding what it already wanted to see.

A few common questions

Why do most A/B tests come back inconclusive? Because most changes genuinely have small or no effect on behaviour, so most tests of those changes show no significant difference. That feels like failure but it isn't, it's the test telling you the truth, that this change doesn't move things meaningfully, which is useful to know. An inconclusive result is a real answer; it just happens to be "this doesn't matter much." The problem only starts when people refuse to accept that answer and go hunting for a winner that isn't there.

What's wrong with stopping a test as soon as it looks significant? It manufactures false positives. A test result fluctuates as data comes in, and with smaller numbers it will very likely cross the significance threshold by pure chance and then cross back. If you stop the instant it looks significant, you're cherry-picking a random high point, and you'll declare "winners" that are really noise far more often than the statistics imply, wins that evaporate when you ship them. The fix is to decide the sample size or duration in advance and not stop early; significance only counts at the planned endpoint, not the first time it flashes green.

If version B is slightly ahead but not significant, can I ship it? Not on that basis. "Not significant" means you can't distinguish the result from no effect at all, so "slightly ahead but not significant" is "we don't know," not "a small win", that slight lead is exactly the kind of difference random noise produces. Shipping B because it was marginally ahead in a flat test is shipping on hope with a chart attached. You can ship B for other reasons (it's simpler or cheaper to maintain, and the test showed it doesn't cost you conversions), but not because you've talked yourself into a win that isn't there.

Is an inconclusive test a waste? No, if you handle it honestly. It tells you the change doesn't matter much, so you stop investing in it and move to a bigger lever. It frees you to choose on other grounds, if both versions perform the same, ship whichever is simpler or cheaper. And it sharpens your next hypothesis, since your reasoning about why it would win was apparently wrong, which is new information about your customers. A flat result handled honestly moves you forward; one tortured into a fake winner sends you confidently backward.