Statistics & sample size
Statistics is the discipline of not fooling yourself — and most reported CRO wins wouldn’t survive it. This module covers the unforgiving few: significance and power, sizing tests from a realistic MDE, why peeking manufactures false wins (and how sequential testing fixes it), the Bayesian alternative, multiple comparisons, variance reduction with CUPED, and the SRM check that voids broken tests.
What you will learn
Why the statistics decide whether you fooled yourself
Statistics is the discipline of not fooling yourself. A/B testing generates random variation, and without the right statistical rules you will repeatedly mistake noise for a winner — ship changes that did nothing, and bank ‘learnings’ that are false. The core ideas are few: significance (is the difference likely real?), power and sample size (could the test even detect a real effect?), and validity checks (did the experiment run correctly?). Get these right and your wins are real; get them wrong and your whole program is confidently lying to you.
Getting numbers is easy; getting numbers you can trust is hard.
The uncomfortable truth: most CRO ‘wins’ reported in case studies wouldn’t survive proper statistics — they were under-powered, peeked-at, or cherry-picked. This module is the immune system that protects your program from its own enthusiasm. You don’t need to be a statistician, but you do need to respect a handful of rules that the math is unforgiving about.
Frequentist testing: p-values and power
The standard A/B test is a frequentist hypothesis test. The p-value is the probability of seeing a difference this large if there were truly no effect; below your threshold (commonly 0.05) you call it ‘significant.’ Statistical power is the chance of detecting a real effect if one exists (commonly targeted at 80%). The two error types: a false positive (calling a non-effect a win) and a false negative (missing a real win). Significance controls the first; power controls the second.
The crucial misreading to avoid: a p-value is not the probability your variant is better, and ‘not significant’ does not mean ‘no difference’ — it often means ‘not enough data to tell.’ Confidence intervals are more honest than a single p-value because they show the range of plausible effects. Report the interval, not just ‘significant/not,’ and you’ll make far better ship/kill decisions.
Sample size, MDE, and power
Before launching, calculate the sample size needed, which depends on your baseline conversion rate, the minimum detectable effect (MDE) you care about, and your chosen significance and power. The smaller the effect you want to detect, the more traffic you need — detecting a 2% relative lift can need many times the sample of a 10% lift. Under-powered tests are the silent killers of CRO: they can’t detect realistic effects, so they come back ‘flat’ and you wrongly conclude nothing worked.
Claim: Required sample size rises sharply as the minimum detectable effect shrinks — halving the MDE roughly quadruples the traffic needed — so low-traffic sites can only reliably detect large effects. Source: Standard power analysis (Evan Miller / Kohavi et al.). Context: Set a realistic MDE for your traffic; if you can’t power a small effect, test bigger swings or accept you can only detect big wins.
Low-traffic teams run test after test that all come back ‘inconclusive’ and conclude experimentation doesn’t work for them. The real problem is they’re trying to detect a 3% lift with traffic that can only ever see a 15% one.
So before committing, I run the math backward: given our traffic and a reasonable run-time, what’s the smallest effect we could actually detect? If that number is bigger than the lift we realistically expect, the test is doomed before launch — and we either swing bigger, test higher up the funnel, or don’t run it.
Knowing your floor for detectable effect turns ‘why do our tests never win’ into ‘we were never powered to see a win.’
Sequential testing and the peeking problem
In a fixed-horizon test, peeking — checking results repeatedly and stopping when you hit significance — dramatically inflates false positives, because with enough looks, random noise will cross the threshold by chance. The fix is either discipline (set the end date and don’t stop early) or proper sequential testing methods (always-valid p-values, group sequential designs) that are mathematically built to let you monitor continuously and stop early without inflating error. Don’t mix the two: peeking at a fixed-horizon test is how most false wins are born.
Claim: Repeatedly peeking at a fixed-horizon A/B test and stopping at the first significant result can inflate the false-positive rate several-fold above the nominal 5%. Source: Online-experimentation statistics literature (Kohavi; Johari et al. on always-valid inference). Context: Either hold to a pre-set sample size, or use a sequential-testing method designed for continuous monitoring — never peek-and-stop on a fixed test.
The Bayesian alternative
Bayesian A/B testing reframes the question from ‘is the difference significant?’ to ‘what’s the probability B beats A, and by how much?’ It outputs intuitive statements like ‘92% probability the variant is better, with an expected lift of 4%’ and an ‘expected loss’ if you choose wrong. It’s not magically immune to peeking and still needs enough data, but many teams find its outputs easier to reason about and act on than p-values. Several major testing tools default to it.
The honest framing: Bayesian vs frequentist is less a war than a choice of language. Bayesian methods give decision-friendly probabilities and naturally support stopping rules based on expected loss; frequentist methods give the familiar significance framework and clearer guarantees about long-run error rates. Pick one, understand its assumptions, and apply it consistently — the danger is mixing frameworks or using either one’s outputs without understanding what they actually claim.
The subtle cousin of peeking is HARKing — Hypothesizing After the Results are Known. You run a test, find an unexpected segment that ‘won,’ and write the hypothesis to match. It feels like insight; it’s a false positive with a story.
I pre-register the primary metric, the segments I’ll analyze, and the decision rule before launch — in writing. Anything I find afterward that I didn’t pre-register is labeled exploratory and must be confirmed in a fresh test, not shipped.
Pre-registration draws a hard line between what the test confirmed and what it merely suggested — the difference between knowledge and a coincidence you fell for.
Multiple comparisons
Every extra metric or variant you test is another chance for a false positive. Test one variant on twenty metrics at p<0.05 and you’d expect about one ‘significant’ result by pure chance even if nothing works. This is the multiple-comparisons problem, and the defenses are: declare one primary metric in advance (the cleanest fix), and where you genuinely test many things, apply a correction (Bonferroni, Benjamini-Hochberg) that tightens the threshold. The cheapest protection is simply not fishing across metrics for a winner.
Teams add ‘just one more variant’ and ‘a few more metrics to watch’ as if they’re free. Each one quietly raises the odds that something looks significant by chance — you’re buying false positives without noticing.
I keep the primary metric singular and the variant count honest, and when a test genuinely needs many arms or metrics, I apply a correction so the threshold tightens to match. The error budget is real; I spend it deliberately.
More things measured isn’t more insight — past a point it’s just more ways to fool yourself.
Variance reduction (CUPED)
Variance reduction techniques — most notably CUPED (Controlled-experiment Using Pre-Existing Data) — use pre-experiment data about each user to strip out predictable noise, so you can detect the same effect with less traffic or in less time. It’s one of the highest-leverage advanced techniques in experimentation: companies running at scale use it to cut required sample sizes substantially. You won’t implement it on day one, but knowing it exists tells you that ‘not enough traffic’ sometimes has a statistical answer, not just ‘wait longer.’
Claim: CUPED uses pre-experiment metrics as covariates to reduce variance, often cutting the sample size needed to detect an effect by a meaningful fraction — effectively buying statistical power without more traffic. Source: Deng, Xu, Kohavi & Walker — CUPED (Microsoft). Context: For traffic-constrained programs, variance reduction is the sophisticated alternative to simply running tests longer.
SRM and validity checks
Before trusting any result, run validity checks — chief among them Sample Ratio Mismatch (SRM): the actual traffic split should match the intended split (a 50/50 test should be ~50/50). A statistically significant deviation means something is broken — a redirect dropping users, a tracking bug, bot contamination — and the result is invalid no matter how good it looks. SRM is the single most important automated guardrail in mature programs: a failed SRM check should void the test, not be explained away.
- What is Sample Ratio Mismatch (SRM)?
- When the observed traffic split differs significantly from the intended split (e.g. you set 50/50 but see 52/48 on large traffic). It signals a bug — redirect loss, tracking error, bot skew — that invalidates the experiment. A failed SRM check should void the result.
- Is a 95% significant result definitely a real win?
- No. It could be a false positive (1-in-20 by chance), the product of peeking, an under-powered fluke, a multiple-comparisons artifact, or an SRM-invalid test. Significance is necessary, not sufficient — check power, peeking, and validity too.
- Do I need to be a statistician to run good tests?
- No, but you must respect a few unforgiving rules: pre-set sample size, don’t peek-and-stop, one primary metric, check SRM, and read confidence intervals not just p-values. Tools handle the math; the discipline is yours.
Where the statistics break
The statistical failure modes: under-powering tests (so real effects look flat), peeking and stopping early (false positives), misreading p-values as ‘probability of being right,’ fishing across many metrics (multiple comparisons), ignoring SRM and validity, and mixing statistical frameworks. Every one produces confident, wrong conclusions — the most dangerous output a program can have.
Too little traffic for the effect size means real wins read as ‘flat.’
Repeated looks at a fixed-horizon test inflate false positives.
Treating p as ‘probability the variant is better’ leads to wrong calls.
Twenty metrics guarantee a chance ‘winner’ (multiple comparisons).
A broken split or tracking bug makes any result meaningless.
Your statistics checklist
Trustworthy results require a few unforgiving habits. Tick what you genuinely do.
Prove it. Earn your passcode.
Ten questions, CASE method (Context · Analysis · Strategy · Execution). Pass at 90% to unlock this module’s completion passcode — retake as many times as you like.