A/B Test Sample Size Calculator
Run a test on too few visitors and a lucky week looks like a winner. This calculator tells you, before you launch, exactly how many visitors each variation needs — so you commit to a test you can actually finish.
Sample size answers one question: how many visitors must each variation collect before a real difference can be told apart from noise? It comes from the standard two-proportion z-test driven by four numbers — your baseline rate, the minimum detectable effect (MDE), your confidence level, and statistical power. Lower baselines and smaller lifts demand far larger samples; required size scales with roughly the inverse square of the effect, so halving the lift you chase quadruples the visitors you must buy.
A/B Test Sample Size Calculator inputs and result
How to use this calculator
- Enter the control’s baseline rateUse the variation you are testing against — conversion rate for a landing page, click rate for an ad. The lower this rate, the bigger the sample you will need.
- Name the minimum detectable effectDecide the smallest relative lift worth detecting. This single choice drives most of the sample size, so pick it on business value, not statistical convenience.
- Set variations, confidence, and powerTwo cells is a classic A/B; each extra challenger adds another full sample. Keep 95% confidence and 80% power unless you have a reason to change them.
- Read the per-variation and total figuresThe big number is visitors per variation; the total multiplies it by your number of cells. That total is what your traffic and timeline must supply.
- Check the confidence table, then exportSee how 90%, 95%, and 99% change the cost before you commit. Copy a share link, download the CSV, or print the plan.
RGM Expert Says
The single most common reason a test ‘fails’ is that it was never powered to succeed. A team ships a variation, watches a noisy dashboard for ten days, sees the lines cross, and calls a winner that evaporates on rollout. We size the test first so that does not happen — the number this tool returns is the price of an answer you can trust, and it is far cheaper to know it before launch than to relearn it after a bad decision.
What surprises clients most is how punishing small lifts are. Going from a 20% detectable effect to a 10% one does not double the sample — it roughly quadruples it, because the math scales with the inverse square of the effect. That is why we push hard on the MDE conversation up front: the honest question is not ‘how small a lift can we find?’ but ‘how big a lift is worth the traffic it would take to prove?’
We also use the figure as a go/no-go filter. If a fair test would need three months of your entire traffic, that is not a sample size problem to solve with patience — it is a signal to test a blunter change, raise the baseline first, or pick a different bet. The calculator turns a vague worry into a number a team can plan a roadmap around.
How it works
Sample size comes from inverting the two-proportion z-test: how many observations per cell are needed so a true difference of your chosen size would clear the significance bar with the power you set.
- p₁ — baseline rate of the control.
- p₂ — target rate after the minimum detectable lift.
- zα — critical value for your confidence (1.96 at 95%, two-sided).
- zβ — value for your power (0.842 at 80%).
This is the standard two-proportion sample-size formula (Wang & Chow, 2007; see this worked reference). For the theory behind power and sample size, see Evan Miller’s sample-size note.
An underpowered test is worse than no test
A test that is too small does not just fail to find an effect — it actively misleads. With too few visitors, ordinary week-to-week variance swamps any real signal, so the result that happens to look biggest gets shipped, and the gain disappears in production. Naming the sample size first is how you avoid paying for noise dressed up as insight.
The MDE is the lever that matters. Because required size scales with roughly the inverse square of the effect, the difference between sizing for a 20% lift and a 10% lift is roughly fourfold. Teams that skip this step tend to discover, three weeks in, that the test would need to run for a quarter — a problem far cheaper to learn about on a calculator than on a calendar.
Sample size is also a roadmap tool. Once you know what a fair test costs in traffic, you can sequence experiments by feasibility: run the high-traffic, big-swing bets that resolve fast, and batch the subtle ones for pages that actually have the volume to settle them.
How baseline and MDE move the sample
There is no single ‘right’ sample size — it falls out of your baseline and the lift you want to detect. These figures (95% confidence, 80% power, per variation) show the shape of the trade-off.
| Baseline | MDE (relative) | Per variation | Read |
|---|---|---|---|
| 5% | 20% | ~8,200 | Workable for most paid funnels |
| 5% | 10% | ~31,000 | Roughly 4× the 20% case |
| 2% | 20% | ~21,000 | Low baselines cost dearly |
| 10% | 20% | ~3,800 | High baselines settle fast |
What the experts say about sizing
Deciding the sample size in advance, and not peeking until you reach it, is the difference between an experiment and a guess dressed up with numbers.
Most A/B tests are underpowered, so most reported wins are overstated; design for the effect you can afford to detect.