--- title: Statistical Significance — Calculation Guide for Marketers | RGM® url: https://realgrowthmatters.com/learn/measurement/statistical-significance-calculation-comprehensive/ updated: 2026-06-10 source_html: https://realgrowthmatters.com/learn/measurement/statistical-significance-calculation-comprehensive/ --- # Statistical Significance — How to Calculate, Interpret, and Apply Statistical significance answers: 'Is this result more likely real or random?' The standard tools: p-values, confidence intervals, sample size calculations, and tests appropriate to data type (z-test, t-test, chi-square). The marketer's job isn't to derive these — it's to know which tool fits which decision and when to trust the output. Statistical significance is the working framework for separating real effects from noise. The formal definition: probability of observing this data (or more extreme) if the null hypothesis (no effect) were true. A small p-value (typically <0.05) suggests the data is unlikely under the null hypothesis, so we reject it. ## The most common test — proportions (A/B test) When comparing two conversion rates: use a **two-proportion z-test**. **z = (p₁ − p₂) / √[p̂(1−p̂)(1/n₁ + 1/n₂)]** Where p̂ is the pooled conversion rate = (x₁+x₂)/(n₁+n₂) Example: Variant A has 100 conversions in 2,000 visitors (5.0%). Variant B has 130 in 2,000 (6.5%). Pooled p̂ = 230/4000 = 0.0575. SE = √[0.0575 × 0.9425 × (1/2000+1/2000)] = 0.00735. Z = (0.065 − 0.050)/0.00735 = 2.04. P-value ≈ 0.041 (two-tailed). Significant at α=0.05. Use a chi-square test or Fisher's exact test as alternatives. Use Bayesian methods if you need ongoing peeking. ## Sample size for proportion test Plan sample size BEFORE the test. Common formula for two-proportion test: **n per arm = (Z\_α/2 + Z\_β)² × [p₁(1−p₁) + p₂(1−p₂)] / (p₁−p₂)²** Where Z\_α/2 = 1.96 for 95% confidence, Z\_β = 0.84 for 80% power. Example: baseline 5%, expected lift to 6% (a 20% relative lift). N per arm ≈ 7,400. To detect a 5% absolute lift (from 5% to 10%) would need only ~300 per arm. Most marketing A/B tests fail to reach significance because they're underpowered — too small a sample to detect realistic effect sizes. ## Test for means — t-test For comparing two means (e.g., AOV between groups): **two-sample t-test**. **t = (M₁ − M₂) / √[s²p(1/n₁ + 1/n₂)]** Where s²p is the pooled variance. Use Welch's t-test if variances differ (default in most statistical software). Sample size: roughly n ≈ 16σ²/δ² where σ is SD and δ is detectable difference. For 80% power and 5% significance, multiplier becomes (Z\_α/2 + Z\_β)² ≈ 7.85; n per arm = 7.85 × σ² / δ². ## Test for categorical — chi-square For independence between categorical variables: **chi-square test**. **χ² = Σ[(Observed − Expected)² / Expected]** Degrees of freedom = (rows − 1) × (columns − 1). Use Fisher's exact test for small expected counts (<5). #### RGM Experts Say The single biggest mistake in marketing A/B testing isn't calculation error — it's peeking. Most teams check tests daily; whenever the result hits p<0.05, they declare a winner and stop. This vastly inflates false-positive rates. The honest options: pre-specify sample size and don't stop until you reach it, OR use sequential testing methodology (mSPRT, AGILE) that allows valid early stopping. Anything else corrupts the p-value. ## P-value interpretation guide - **p < 0.001**: Very strong evidence against null; result very unlikely by chance - **p < 0.01**: Strong evidence against null - **p < 0.05**: Conventional significance threshold; some evidence against null - **p < 0.1**: Marginal evidence; treat with caution - **p ≥ 0.1**: Insufficient evidence to reject null - **p > 0.5**: Data consistent with null hypothesis - Important: a non-significant p-value doesn't mean 'no effect' — it means 'didn't detect effect in this test.' Smaller sample sizes simply fail to detect smaller effects. ## Confidence interval calculation - For a proportion: **p̂ ± Z × √[p̂(1−p̂)/n]** - For 95% CI, Z=1.96. For 99% CI, Z=2.576. - Example: 5% conversion rate from 2,000 visitors. SE = √(0.05×0.95/2000) = 0.00487. 95% CI = 0.05 ± 1.96×0.00487 = [4.05%, 5.95%]. - For a mean: **M ± t × (s/√n)** where t is from t-distribution with df = n−1. ## Common pitfalls - **Peeking inflates false positives** — pre-register or use sequential testing - **Multiple testing inflates false positives** — use Bonferroni correction (α/k) or Benjamini-Hochberg for FDR - **Simpson's Paradox** — aggregate trends can reverse in subgroups; always check segments - **Sample ratio mismatch** — uneven traffic splits indicate randomization bug - **Novelty effect** — early test results biased by user reaction to change - **Primacy bias** — opposite of novelty; users prefer familiar variant initially - **Ignoring practical significance** — statistical significance isn't always practically meaningful - **Underpowered tests** — calculate sample size before running ## Free calculators and tools - **Evan Miller's calculators**: evanmiller.org — proportions, sample size, sequential testing - **VWO Significance Calculator**: free A/B test significance - **Optimizely Sample Size Calculator**: visual sample-size planning - **SurveyMonkey Sample Size Calculator**: for surveys - **Statsig**: experimentation platform with built-in significance - **Python (scipy.stats)**: comprehensive statistical functions - **R**: statistical computing standard ## Related guides - See [R² vs p-value](/learn/measurement/r-squared-vs-p-value-regression/) - See [regression analysis](/learn/measurement/regression-analysis-for-marketing/) - See [A/B testing](/learn/concepts/ab-testing-statistical-significance/) ## Sources 1. [1]Evan Miller statistical tools and writings; Wasserman All of Statistics; Kohavi, Tang, Xu Trustworthy Online Controlled Experiments ### Related guides - [R-squared vs p-value](/learn/measurement/r-squared-vs-p-value-regression/) - [Regression analysis](/learn/measurement/regression-analysis-for-marketing/) - [A/B testing significance](/learn/concepts/ab-testing-statistical-significance/)