What does statistical significance actually mean here?

It means the observed difference is unlikely to have arisen from random variation alone, if the two versions were in fact identical. At a p-value of 0.05 that unlikelihood is about one in twenty. It is a statement about how surprising the data would be under the assumption of no effect, not a probability that your variant is better, and not a measure of how large the improvement is.

Can I stop the test as soon as it reaches 95%?

Not without breaking the maths. A test checked repeatedly will cross any threshold eventually through noise alone, so stopping at the first significant reading turns a one in twenty false positive rate into something closer to one in three. Either fix the sample size in advance and run to it, or use a method designed for continuous monitoring, such as a sequential or Bayesian approach.

How many visitors do I need per arm?

It is driven by your baseline conversion rate and the smallest lift worth detecting, and the requirement grows steeply as the lift shrinks. Detecting a 20% relative improvement on a 3% baseline takes a few thousand visitors per arm; detecting a 5% improvement on the same baseline takes tens of thousands. If the honest sample size exceeds the traffic you have, test bigger changes rather than running longer.

The result is significant but the lift is tiny. Should I ship it?

Significance and importance are separate questions, and very large samples routinely produce the first without the second. Convert the absolute lift, in percentage points, into annual money at your current traffic, then weigh that against the cost of maintaining the change. A statistically certain improvement worth a rounding error is a legitimate thing to decline.

Yes. This tool does its work in your browser, so whatever you type, paste or upload stays on your device. Nothing is sent to our servers, which is also why it keeps working if you go offline after the page has loaded.

Yes. Every tool here is free with no account, no credit card and no usage cap. They exist so that the people who need our scheduling product find us, which only works if the tools are genuinely useful on their own.

Do I need an account?

No. Open the page and use it. An account is only for OctoSpark itself, where you plan, schedule and publish a whole calendar rather than fixing one post at a time.

Can I use the output commercially?

Yes. Anything you produce here is yours, including for client and commercial work. We claim no rights over it and we do not watermark it.

OctoSpark

Free tool — runs in your browser

A/B test significance calculator

Calculate relative lift, absolute lift and two-tailed significance for an A/B test from four numbers, with a warning when the sample is too small to hold.

Solve for

Your numbers

Control visitors

Everybody who entered the control arm, converted or not. Enter it a whole number.

Control conversions

Conversions in the control arm, counted on the same definition as the variant. Enter it a whole number.

Variant visitors

Everybody who entered the variant arm over the same period. Enter it a whole number.

Variant conversions

Conversions in the variant arm, on the identical definition. Enter it a whole number.

Relative lift

28.26%

The variant is 28.26% better than the control in relative terms. Check the significance panel before you tell anybody that.

The sum this ran

Relative lift = ((Variant rate - Control rate) ÷ Control rate) × 100

Control rate = (Control conversions ÷ Control visitors) × 100 = 2.3%

Variant rate = (Variant conversions ÷ Variant visitors) × 100 = 2.95%

Not significant at the 95% level

Control 2.3%, variant 2.95%. Two-tailed p-value 6.9%, confidence 93.1%.

The gap between these two arms is within the range ordinary sampling noise produces at these volumes. Keep running to a sample size you fixed in advance rather than stopping at the first encouraging reading.

Everything here runs in your browser. Nothing you type is uploaded, which matters when the inputs are spend, revenue and margin.

A significance number is not a decision

The panel on this page reports a two-tailed p-value from a pooled two-proportion z-test, which is the standard test behind almost every A/B tool. What it tells you is narrower than what people read into it, and the difference matters when real money follows the result.

A p-value below 0.05 means that if the two versions were genuinely identical, a difference this large or larger would turn up in about one test in twenty by chance alone.
It does not mean there is a 95% probability the variant is better. That is a different quantity, and it needs a prior that a frequentist test does not have.
It says nothing about how much better. A huge sample can show overwhelming significance on a lift far too small to be worth the engineering effort of shipping.
The test assumes the visitors were randomly assigned, counted once each, and measured over the same period. Break any of those and the number is arithmetic rather than evidence.

The exact sums this page runs

The lift arithmetic is visible here so it can be checked. The significance panel underneath runs a pooled two-proportion z-test, which compares the observed gap against the spread you would expect from sampling noise alone at these volumes.

Control rate = (Control conversions ÷ Control visitors) × 100. The baseline the variant is measured against.
Variant rate = (Variant conversions ÷ Variant visitors) × 100. The challenger, computed identically so the comparison is fair.
Relative lift = ((Variant rate - Control rate) ÷ Control rate) × 100. The headline. It is a change in a rate, which is why 2.0% to 2.4% is reported as a 20% lift.
Absolute lift in percentage points = Variant rate - Control rate. The honest version for a forecast: it multiplies straight through traffic, where a relative lift does not.
Variant conversion rate = (Variant conversions ÷ Variant visitors) × 100. The variant on its own, for when you want the rate rather than the comparison.

Running the test so the number means something

Most failed experiments fail in the design rather than in the analysis. The sequence below is what separates a result you can act on from a number that will not reproduce.

Decide the sample size before you start, from your baseline rate and the smallest lift that would be worth shipping. Then commit to running to it.
Change one thing. A redesign that alters the headline, the image and the button gives you a result with no explanation and nothing to generalise from.
Run for whole weeks. Weekday and weekend traffic convert differently, and a test stopped on a Thursday is partly measuring the day.
Do not stop early because the number crossed 95%. Peeking repeatedly and stopping at the first significant moment inflates the false positive rate dramatically, often past one in three.
Check that assignment is genuinely random and sticky, so a returning visitor sees the same variant rather than being reassigned and counted twice.
Report the confidence interval alongside the lift, and treat a result that straddles zero as what it is: no evidence either way, which is a legitimate outcome.

What to do with a result that is not significant

Most tests do not reach significance, and that is the normal condition rather than a failure of the tool. The useful response depends on why the test fell short, and there are only a few reasons.

Underpowered: the effect might be real but the sample was never large enough to detect it. Calculate the sample size the observed effect would need, and decide whether you can afford to run that long.
Genuinely no difference: the change did not matter to users. This is a real finding and it is worth recording, because it stops the same idea being retried every eighteen months.
Too small to care about: a lift of a fraction of a percentage point may be real and still not worth the maintenance cost of a second code path.
Broken instrumentation: wildly unequal arm sizes, or conversion counts that do not match your analytics, mean the test should be fixed and rerun rather than interpreted.

The rest of the job

You just fixed one post. Now do the month.

The a/b test significance calculator solves one piece of one post. OctoSpark plans, writes and schedules the whole calendar across every network you use, publishes it for you, and reports what actually performed.

One calendar for every network
Schedule once, publish everywhere
Drafts written in your own voice
See what actually performed

Open the full sign-up page

No card needed. These tools stay free either way.

Frequently asked questions

How this tool works, what it cannot do, and what happens to what you put into it.

What does statistical significance actually mean here?
It means the observed difference is unlikely to have arisen from random variation alone, if the two versions were in fact identical. At a p-value of 0.05 that unlikelihood is about one in twenty. It is a statement about how surprising the data would be under the assumption of no effect, not a probability that your variant is better, and not a measure of how large the improvement is.
Can I stop the test as soon as it reaches 95%?
Not without breaking the maths. A test checked repeatedly will cross any threshold eventually through noise alone, so stopping at the first significant reading turns a one in twenty false positive rate into something closer to one in three. Either fix the sample size in advance and run to it, or use a method designed for continuous monitoring, such as a sequential or Bayesian approach.
How many visitors do I need per arm?
It is driven by your baseline conversion rate and the smallest lift worth detecting, and the requirement grows steeply as the lift shrinks. Detecting a 20% relative improvement on a 3% baseline takes a few thousand visitors per arm; detecting a 5% improvement on the same baseline takes tens of thousands. If the honest sample size exceeds the traffic you have, test bigger changes rather than running longer.
The result is significant but the lift is tiny. Should I ship it?
Significance and importance are separate questions, and very large samples routinely produce the first without the second. Convert the absolute lift, in percentage points, into annual money at your current traffic, then weigh that against the cost of maintaining the change. A statistically certain improvement worth a rounding error is a legitimate thing to decline.
Is my data private?
Yes. This tool does its work in your browser, so whatever you type, paste or upload stays on your device. Nothing is sent to our servers, which is also why it keeps working if you go offline after the page has loaded.
Is this really free?
Yes. Every tool here is free with no account, no credit card and no usage cap. They exist so that the people who need our scheduling product find us, which only works if the tools are genuinely useful on their own.
Do I need an account?
No. Open the page and use it. An account is only for OctoSpark itself, where you plan, schedule and publish a whole calendar rather than fixing one post at a time.
Can I use the output commercially?
Yes. Anything you produce here is yours, including for client and commercial work. We claim no rights over it and we do not watermark it.