--- title: P-value — definition | RGM® Glossary url: https://realgrowthmatters.com/glossary/p-value/ updated: 2026-06-10 source_html: https://realgrowthmatters.com/glossary/p-value/ --- # P-value P-val·ue/pi ˈvælju/noun It is NOT the chance your result is real — and that single misreading has wrecked more decisions than any other in analytics. Term : P-value Measures : Surprise under the null hypothesis Common threshold : p < 0.05 Is NOT : The probability the effect is real ## Forms & parts of speech p < 0.05 · phrase The conventional significance cut. "It cleared **p < 0.05** — but with twenty metrics tested, one was bound to." ## Definition in plain terms A p-value is the probability of observing a result at least as extreme as the one you got, ASSUMING there is no real effect (the null hypothesis is true). A p-value of 0.03 means 'if nothing were actually happening, you'd see a result this strong only 3% of the time' — surprising enough, by the p < 0.05 convention, to call the result statistically significant. Crucially, it does NOT say the effect is 97% likely to be real; that's the single most damaging misreading in applied analytics. ## The mechanics The p-value answers a narrow question (how surprising is this data under 'no effect?') and is routinely asked to answer a different one (how likely is my hypothesis true?) — the inversion fallacy. Its real-world failure modes: P-HACKING (testing many variants or metrics until one crosses 0.05 — with 20 tests, one false positive is expected by chance), PEEKING (checking repeatedly and stopping when significant — which inflates the false-positive rate far above 5%), and confusing statistical significance with PRACTICAL significance (a real but trivially small effect can be 'significant' at huge sample sizes). The American Statistical Association's 2016 statement formally warned against exactly these abuses. ## When it matters The p-value matters as one input to a decision, never the decision itself — read alongside the effect SIZE (is it big enough to matter?), the confidence interval (how precise?), prior plausibility, and whether the test was pre-registered or fished. For marketers the discipline is cultural: a team that ships on 'p < 0.05' alone will ship noise, while a team that asks 'how big, how precise, how surprising, and did we go looking for it?' makes durable decisions. Significance is a smoke alarm, not a verdict. **Worked example.** A CRO team celebrates a 'significant' winner (p = 0.04) and rolls it out — and the lift vanishes in production. The post-mortem finds the rot: the test tracked nine metrics, and the team reported the one that crossed 0.05 (p-hacking by metric-shopping), on a test they'd peeked at daily and stopped the moment it went green. The rebuild installs the rigor: one pre-declared primary metric, a fixed sample size run to completion, and decisions that weigh effect size and confidence interval alongside the p-value. Win rate on tests drops — and the wins that remain actually replicate. **Failure modes to watch.** Reading the p-value as the probability the effect is real; p-hacking by testing many metrics or variants; peeking and stopping at significance; and confusing statistical significance with a practically meaningful effect. ## Synonyms & antonyms ### Synonyms p-valueprobability value ### Antonyms practical significance (the size question)p-hacking (the abuse) ## Origin & history The p-value was popularized by Ronald Fisher in *Statistical Methods for Research Workers* (1925), where he proposed the 5% threshold as a convenient (not sacred) convention; the Neyman-Pearson framework added the formal error-rate machinery, and the ASA's 2016 statement codified the modern warnings against its misuse. Etymology: [source](https://en.wikipedia.org/wiki/P-value). ## Usage trends Search interest for this term over the last five years: [View interest-over-time on Google Trends →](https://trends.google.com/trends/explore?q=p%20value&date=today%205-y) ## Common questions What is a p-value? : The probability of seeing a result at least as extreme as yours if there were no real effect — a measure of surprise under the null. What does a p-value NOT tell you? : It is not the probability your hypothesis is true, nor the probability the effect is real — that inversion is the common fatal error. What is p-hacking? : Testing many variants or metrics until one crosses the significance threshold — manufacturing false positives by chance. ## Related tools & calculators - tool[Experiment planner](/tools/experiment-planner/) - tool[Funnel drop-off analyzer](/tools/funnel-drop-off-analyzer/) ## Resources & people to follow - referenceASA Statement on p-values (2016) - book*Trustworthy Online Controlled Experiments* — Kohavi, Tang & Xu - referenceRGM analysis — read size, precision, and intent alongside p Curated, non-competitor resources verified per term. ## Related training - module[CRO & experimentation](/training/cro-experimentation/) ## Disciplines Areas of marketing where p-value is a core concern: [Experimentation](/training/cro-experimentation/)[Analytics](/training/marketing-analytics/) ## Read next ## Related terms [Statistical significance](/glossary/statistical-significance/)[Statistical power](/glossary/statistical-power/)[Sample size](/glossary/sample-size/)[A/B test](/glossary/ab-test/)[Ronny Kohavi](/glossary/ronny-kohavi/) ## Sources 1. trends[Google Trends — "p value"](https://trends.google.com/trends/explore?q=p%20value&date=today%205-y)