Statistical Significance — How to Calculate, Interpret, and Apply

Statistical significance answers: 'Is this result more likely real or random?' The standard tools: p-values, confidence intervals, sample size calculations, and tests appropriate to data type (z-test, t-test, chi-square). The marketer's job isn't to derive these — it's to know which tool fits which decision and when to trust the output.

Statistical significance is the working framework for separating real effects from noise. The formal definition: probability of observing this data (or more extreme) if the null hypothesis (no effect) were true. A small p-value (typically <0.05) suggests the data is unlikely under the null hypothesis, so we reject it.

The most common test — proportions (A/B test)

When comparing two conversion rates: use a two-proportion z-test.

z = (p₁ − p₂) / √[p̂(1−p̂)(1/n₁ + 1/n₂)]

Where p̂ is the pooled conversion rate = (x₁+x₂)/(n₁+n₂)

Example: Variant A has 100 conversions in 2,000 visitors (5.0%). Variant B has 130 in 2,000 (6.5%). Pooled p̂ = 230/4000 = 0.0575. SE = √[0.0575 × 0.9425 × (1/2000+1/2000)] = 0.00735. Z = (0.065 − 0.050)/0.00735 = 2.04. P-value ≈ 0.041 (two-tailed). Significant at α=0.05.

Use a chi-square test or Fisher's exact test as alternatives. Use Bayesian methods if you need ongoing peeking.

Sample size for proportion test

Plan sample size BEFORE the test. Common formula for two-proportion test:

n per arm = (Z_α/2 + Z_β)² × [p₁(1−p₁) + p₂(1−p₂)] / (p₁−p₂)²

Where Z_α/2 = 1.96 for 95% confidence, Z_β = 0.84 for 80% power.

Example: baseline 5%, expected lift to 6% (a 20% relative lift). N per arm ≈ 7,400. To detect a 5% absolute lift (from 5% to 10%) would need only ~300 per arm.

Most marketing A/B tests fail to reach significance because they're underpowered — too small a sample to detect realistic effect sizes.

Test for means — t-test

For comparing two means (e.g., AOV between groups): two-sample t-test.

t = (M₁ − M₂) / √[s²p(1/n₁ + 1/n₂)]

Where s²p is the pooled variance. Use Welch's t-test if variances differ (default in most statistical software).

Sample size: roughly n ≈ 16σ²/δ² where σ is SD and δ is detectable difference. For 80% power and 5% significance, multiplier becomes (Z_α/2 + Z_β)² ≈ 7.85; n per arm = 7.85 × σ² / δ².

Test for categorical — chi-square

For independence between categorical variables: chi-square test.

χ² = Σ[(Observed − Expected)² / Expected]

Degrees of freedom = (rows − 1) × (columns − 1).

Use Fisher's exact test for small expected counts (<5).

RGM Experts Say

The single biggest mistake in marketing A/B testing isn't calculation error — it's peeking. Most teams check tests daily; whenever the result hits p<0.05, they declare a winner and stop. This vastly inflates false-positive rates. The honest options: pre-specify sample size and don't stop until you reach it, OR use sequential testing methodology (mSPRT, AGILE) that allows valid early stopping. Anything else corrupts the p-value.

P-value interpretation guide

  • p < 0.001: Very strong evidence against null; result very unlikely by chance
  • p < 0.01: Strong evidence against null
  • p < 0.05: Conventional significance threshold; some evidence against null
  • p < 0.1: Marginal evidence; treat with caution
  • p ≥ 0.1: Insufficient evidence to reject null
  • p > 0.5: Data consistent with null hypothesis
  • Important: a non-significant p-value doesn't mean 'no effect' — it means 'didn't detect effect in this test.' Smaller sample sizes simply fail to detect smaller effects.

Confidence interval calculation

  • For a proportion: p̂ ± Z × √[p̂(1−p̂)/n]
  • For 95% CI, Z=1.96. For 99% CI, Z=2.576.
  • Example: 5% conversion rate from 2,000 visitors. SE = √(0.05×0.95/2000) = 0.00487. 95% CI = 0.05 ± 1.96×0.00487 = [4.05%, 5.95%].
  • For a mean: M ± t × (s/√n) where t is from t-distribution with df = n−1.

Common pitfalls

  • Peeking inflates false positives — pre-register or use sequential testing
  • Multiple testing inflates false positives — use Bonferroni correction (α/k) or Benjamini-Hochberg for FDR
  • Simpson's Paradox — aggregate trends can reverse in subgroups; always check segments
  • Sample ratio mismatch — uneven traffic splits indicate randomization bug
  • Novelty effect — early test results biased by user reaction to change
  • Primacy bias — opposite of novelty; users prefer familiar variant initially
  • Ignoring practical significance — statistical significance isn't always practically meaningful
  • Underpowered tests — calculate sample size before running

Free calculators and tools

  • Evan Miller's calculators: evanmiller.org — proportions, sample size, sequential testing
  • VWO Significance Calculator: free A/B test significance
  • Optimizely Sample Size Calculator: visual sample-size planning
  • SurveyMonkey Sample Size Calculator: for surveys
  • Statsig: experimentation platform with built-in significance
  • Python (scipy.stats): comprehensive statistical functions
  • R: statistical computing standard

Related guides

Sources

  1. [1]Evan Miller statistical tools and writings; Wasserman All of Statistics; Kohavi, Tang, Xu Trustworthy Online Controlled Experiments