Type II Error

A Type II error is a false negative — the mistake of failing to detect an effect that is genuinely there. In the language of hypothesis testing, it means failing to reject the null hypothesis, the assumption of 'no difference,' when the null is actually false. Picture an A/B test where variant B truly does convert better than A. If the test, thrown off by too little data or too much noise, reports 'no significant difference,' the team keeps the weaker version and never knows what it left on the table. That miss is a Type II error. Its probability is written as beta (β), and the complement of beta — one minus beta — is statistical power, the chance the test correctly catches a real effect. So a high Type II error rate and low power are two names for the same weakness: a test that cannot see what is in front of it.

What makes Type II errors dangerous is that they are silent. A false positive announces itself — you ship a change, it fails to deliver, and eventually someone notices. A false negative leaves no trace: the winning idea is quietly shelved, the effective channel is judged useless, the profitable segment is dismissed, and the business simply moves on, poorer and unaware. Underpowered experiments are the usual culprit. A test with too few visitors, run for too short a time, or aimed at an effect smaller than it can resolve will miss real differences again and again, and every miss looks like a clean 'no effect' result. In marketing, where many true improvements are modest single-digit lifts, the cost of Type II errors compounds: a steady stream of real wins gets declared null, and the program stops improving without anyone able to point to why.

Type II and Type I errors are the two opposite ways a test can be wrong, and confusing them is a classic mistake. A Type I error is a false positive: rejecting a true null hypothesis, concluding there is an effect when there is none — declaring a variant a winner that is really no better. A Type II error is a false negative: failing to reject a false null, missing an effect that is real — declaring a genuine winner no better than the control. Type I is the false alarm; Type II is the missed detection. Their probabilities have their own symbols: alpha (α) for Type I, which you set as the significance level, often 0.05, and beta (β) for Type II. One test can only make one of these mistakes on a given decision — either it wrongly cries wolf or it wrongly stays silent — but a program makes both over time.

The two errors are linked by a tradeoff you cannot wish away. Make a test stricter — lower alpha so it demands stronger evidence before declaring an effect — and you cut false positives while raising false negatives, because a cautious test also misses more real effects. Loosen it and the balance tips the other way. The only way to reduce both errors at once is to feed the test more information: a larger sample, a cleaner design, or a longer run. That is why sample size is the real lever. Which error to fear more depends on the stakes. In medicine, a Type I error, approving a drug that does not work, is often the graver risk. In fast-moving marketing, a Type II error — killing a real improvement — is frequently the costlier one, because missed growth compounds. Naming which error you are guarding against should come before you set the threshold.

Avoiding Type II errors is mostly a matter of power, and power is mostly a matter of planning. Before running a test, do a power analysis: decide the smallest effect worth detecting, then work out the sample size needed to catch it with, say, 80 or 90 percent power at your chosen significance level. Run the test to that size and for its full planned duration, rather than stopping the moment things look flat. Reduce noise where you can — cleaner metrics, tighter targeting, paired or blocked designs — because less noise means more power for the same sample. And when a test does come back null, read it correctly: 'we failed to detect an effect' is not the same as 'there is no effect.' A wide confidence interval around zero is a shrug, not a verdict, and often means the test was simply too small to conclude anything.

The failures are almost all failures of power dressed up as findings. Teams run underpowered tests, get a null result, and announce that a channel, a creative, or a segment 'does not work,' when the truth is the test never had the sensitivity to tell. They chase tiny effects with samples fit only for huge ones. They stop early on a flat-looking curve, guaranteeing misses. And they treat 'not significant' as 'no difference,' burying real effects under a statistical technicality. The remedy is to design for the effect you care about, size the test to detect it, resist premature calls, and report a null as the uncertain, power-limited statement it usually is. A Type II error is the quiet cost of asking a question without giving the test enough evidence to answer — and in a growth program, the missed wins add up fast.

The Type I and Type II error framework was formalized in the 1920s and 1930s by statisticians Jerzy Neyman and Egon Pearson as part of their theory of hypothesis testing.

Etymology: source.

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Areas of marketing where type ii error is a core concern: