Hypothesis Testing

Hypothesis testing is the statistical framework behind every test verdict marketing issues: assume the boring explanation — the null hypothesis, 'no real effect, any difference is chance' — then compute how surprising the observed data would be under that assumption. Surprising enough (the P-VALUE below a pre-chosen threshold) and you reject the null; otherwise you don't — and the framework's entire value depends on decisions made BEFORE the data arrives, which is precisely where practice keeps breaking it.

The moving parts, in the order they should be fixed: the hypothesis itself (specific and directional — 'the new checkout raises completion rate' — written before the test, because a hypothesis chosen after seeing the data is a description wearing a lab coat), the metric (ONE primary, named in advance; secondary metrics explore, they don't decide), the SAMPLE-SIZE (powered for the minimum effect worth detecting — the underpowered test is the genre's quiet tragedy, designed to miss the very effects it hunts), the significance threshold (alpha, conventionally 0.05 — a false-alarm budget, not a truth oracle), and only then the data. The two errors the design trades between: TYPE-I-ERROR (rejecting a true null — shipping a mirage) and Type II (missing a real effect — the cost of underpowering), with the trade governed by stakes: a cheap reversible change can tolerate more false alarms than a pricing migration. The violations that quietly dominate practice, each with its own glossary entry because each is an industry: peeking (checking significance repeatedly and stopping at the first happy reading — inflating false alarms far past the stated alpha), the MULTIPLE-COMPARISONS harvest (twelve metrics, twenty segments, one 'win' guaranteed by arithmetic), HARKing (hypothesizing after results are known), and the silent file drawer of unreported losers. And the interpretation honesty the p-value entry expands: significance is not size (a trivial effect turns significant at scale), non-significance is not 'no effect' (often just 'no power'), and the p-value never states the probability the hypothesis is true — it states the probability of the data, assuming it isn't.

Hypothesis testing matters wherever marketing claims causality from comparison — A/B tests, geo experiments, creative tests, the email subject-line ritual — as the grammar that separates evidence from anecdote-at-scale. It matters most as pre-commitment (hypothesis, metric, n, threshold, stopping rule — written down first), and least as decoration, where the vocabulary dresses up peeked, harvested, underpowered comparisons. The discipline is the boring checklist: one pre-registered primary metric, power computed for the effect worth caring about, no peeking without sequential corrections, all results logged including the losers - because the framework only means what its pre-commitments make it mean.

The framework fused two twentieth-century inventions - Fisher's significance testing (the p-value, 1925) and Neyman-Pearson's decision framework (the two errors, alpha, power, 1933) - into the hybrid every A/B platform ships; marketing inherited the machinery, the conventions, and the misreadings as a set.

Etymology: source.

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Areas of marketing where hypothesis testing is a core concern: