Statistical Significance Calculator
Your variant is up — but is it really winning, or did you just catch a good week? Enter the visitors and conversions for each group and find out whether the difference is real or noise.
Statistical significance asks one thing: if the control and the variant were truly identical, how often would you see a gap this big by pure chance? This calculator computes each conversion rate, the z-statistic from a pooled two-proportion test, and the two-tailed p-value. A small p-value (under 0.05 for 95% confidence) means the difference is unlikely to be a fluke; a large one means you are probably looking at noise and should keep collecting.
Statistical Significance Calculator inputs and result
How to use this calculator
- Enter the control groupPut the control’s total visitors and its conversions. These define the baseline rate the variant is measured against.
- Enter the variant groupAdd the variant’s visitors and conversions. The two denominators do not need to match, but very lopsided splits are harder to read.
- Pick your confidence level95% is the default. Choose 99% for high-stakes decisions where a false positive is expensive, or 90% only for low-risk, directional reads.
- Read the p-value and verdictThe p-value is the chance of seeing this gap if the versions were identical. Below your threshold, the verdict reads significant; above it, the difference is not yet distinguishable from noise.
- Confirm the test was pre-sizedSignificance is only trustworthy if you reached a planned sample and did not stop the moment it crossed. Then export the result with the share, CSV, or print buttons.
RGM Expert Says
A significance calculator is the last gate before a decision, and it is where a lot of money is saved or lost. The temptation is to open the dashboard daily and stop the second the variant looks like a winner — which, statistically, guarantees you will eventually see a ‘winner’ that is not one. We treat this tool as a check against a pre-committed sample, not a green light to peek.
The number people misread most is the p-value. It is not the probability the variant is better; it is the probability of seeing a gap this large if the two were actually the same. A p-value of 0.04 does not mean a 96% chance the variant wins — it means a difference this big would show up about 4% of the time by chance alone. Keeping that distinction straight is half of running honest experiments.
We also watch for the opposite error: declaring ‘no difference’ from a test that never had the sample to find one. A non-significant result on a tiny test is not evidence the variant failed — it is evidence you did not look hard enough. We pair this calculator with a sample-size check so a null result actually means something.
How it works
The test pools both groups under the assumption they are identical, measures how many standard errors apart the observed rates are, and converts that distance into a two-tailed p-value using the normal distribution.
- p₁, p₂ — conversion rates of control and variant (x/n).
- p̅ — pooled rate under the null hypothesis that the two are equal.
- Φ — standard normal CDF; we use the Abramowitz-Stegun erf approximation.
- p-value — two-tailed; the chance of a gap this large under the null.
The pooled two-proportion z-test is the standard frequentist significance test for comparing two rates. The two-tailed p-value uses the normal CDF (Abramowitz-Stegun erf). For the reasoning behind significance and peeking, see Georgiev, Analytics-Toolkit and Optimizely’s glossary.
Significance protects you from acting on noise
Conversion rates wobble. On a normal day, the same page can post 4.6% and 5.4% in two halves of the week with nothing changed. Significance testing is what separates a real improvement from that ordinary wobble — it tells you how surprising your observed gap would be if the variant changed nothing at all.
The most expensive mistake is peeking: checking the result repeatedly and stopping at the first moment it crosses 95%. Each look is another chance to catch a random high, so a test that peeks daily can hit ‘significance’ far more than 5% of the time even when the variant is worthless. The fix is to set a sample size in advance and judge the result once.
Significance is necessary but not sufficient. A statistically significant 0.1% lift may not be worth the engineering to ship it. Read the p-value alongside the size of the effect and a confidence interval, so you act on changes that are both real and material.
How to read your p-value
The p-value is the probability of seeing a difference at least this large if the two versions were truly identical. Smaller is stronger evidence against ‘no difference’.
| p-value | Confidence | How to treat it |
|---|---|---|
| Above 0.10 | Below 90% | Inconclusive — keep collecting |
| 0.05 to 0.10 | 90% to 95% | Suggestive, not decisive |
| 0.01 to 0.05 | 95% to 99% | Significant for most decisions |
| Below 0.01 | Above 99% | Strong evidence |
What the experts say about significance
A p-value is the probability of the data given the null hypothesis — not the probability the hypothesis is true. Conflating the two is the most common error in A/B testing.
The single biggest mistake in online experiments is stopping the test the moment it looks significant. Pre-commit to the sample, then judge once.