Bayesian Decision Making

Why Bayesian is gaining adoption. Mental models vs frequentist, priors, interpretation, sequential testing, MMM, and decision frameworks.

Why Bayesian is gaining adoption

Frequentist statistics produces outputs like p-values that stakeholders misinterpret. Bayesian methods produce outputs like "87% probability variant beats control with expected lift 8–15%" that align more naturally with decision-making.

Beyond communication, Bayesian methods handle sequential testing, prior knowledge integration, and small samples better than frequentist. Modern tools (Statsig, Eppo, Meta Robyn) increasingly default to Bayesian approaches.

Bayesian vs frequentist

Aspect	Frequentist	Bayesian
Probability concept	Long-run frequency	Degree of belief
Parameters	Fixed unknowns	Probability distributions
Output	P-values, confidence intervals	Posterior distributions, credible intervals
Interpretability	Counter-intuitive	More natural
Prior knowledge	Ignored	Explicitly incorporated
Sequential testing	Requires adjustments	Naturally valid
Small samples	Conservative	Better with informative priors

Priors

• Prior: What you believe before seeing the data.
• Likelihood: How likely is the data given parameter values?
• Posterior: Updated belief after seeing data; combines prior and likelihood.

Where priors come from

• Historical data. Past similar tests provide informative priors.
• Industry benchmarks. Conversion rates in your category.
• Expert judgment. Domain expertise quantified.
• Weakly informative (default). Light prior assumptions when knowledge is limited.
• Non-informative / uniform. No prior knowledge assumed.

Common concerns about priors

• Subjectivity. Different priors produce different posteriors.
• Solution: Sensitivity analysis with multiple priors; report range.
• Solution: Use weakly informative defaults; let data dominate posterior.
• Strong priors: Use when data is limited and prior knowledge is reliable.

Bayesian interpretation

Common Bayesian outputs:

• Probability that variant beats control. "87% probability of positive lift."
• Expected lift with credible interval. "Expected 5.2% lift, 95% credible interval 1.8–9.1%."
• Probability of meeting threshold. "72% probability of >3% lift."
• Posterior distribution. Full distribution of plausible effect sizes.

Bayesian sequential testing

• Probability of effect updates naturally as data accumulates.
• No false-positive inflation from sequential checking.
• Allows early stopping when probability crosses decision threshold.
• Allows continuation when results are ambiguous.
• Modern tools (Statsig, Eppo) implement this directly.

Bayesian MMM

• Modern MMM is largely Bayesian.
• Priors set on coefficients based on historical knowledge or experiments.
• Posterior distributions quantify uncertainty in channel contribution.
• Calibration with incrementality experiments straightforward.
• Tools: Meta Robyn, Google LightweightMMM, PyMC, Stan.

From probability to decision

• Loss function thinking. What's the cost of shipping a non-winner vs not shipping a winner?
• Asymmetric decisions. If the downside of shipping a bad variant is small (easy to revert), ship at lower probability threshold.
• Expected value calculations. Probability × impact; informs investment decisions.
• Risk-adjusted decisions. Bayesian credible intervals enable proper risk weighting.

Tools and frameworks

• Experimentation platforms. Statsig, Eppo (Bayesian by default); Optimizely Stats Engine (always-valid p-values, Bayesian-influenced).
• MMM platforms. Meta Robyn, Google LightweightMMM, Uber Orbit, Recast, Haus.
• General-purpose: PyMC, Stan, NumPyro (Python); brms, rstanarm (R).
• Cloud: Google BigQuery ML, AWS SageMaker.

Advanced playbook

• Weakly informative defaults. Start with weak priors; let data drive.
• Prior sensitivity analysis. Multiple priors; report range; transparent.
• Decision thresholds documented. Probability thresholds for "ship" vs "kill" decisions.
• Bayesian sequential by default. Modern experimentation tooling enables this; use it.
• MMM with experiment-informed priors. The Bayesian-experiment hybrid is best-in-class.
• Credible intervals over confidence intervals in reporting. More natural for stakeholders.
• Decision-theoretic framing. Loss functions; expected value calculations.
• Calibration with reality. Compare Bayesian predictions to actual outcomes over time.
• Tooling investment. PyMC, Stan, NumPyro for in-house teams; vendor tools for less custom needs.
• Methodology transparency. Stakeholders understand what's Bayesian and why.

Common mistakes

• Subjective priors masquerading as objective; sensitivity not tested.
• Bayesian outputs misinterpreted as frequentist p-values.
• Strong priors with limited data; posterior driven by prior.
• Weakly informative priors when strong knowledge exists; informative priors would improve.
• No decision thresholds documented; ambiguous outcomes.
• Posterior distributions reported without probability summaries; stakeholders confused.
• Bayesian MMM without experiment calibration; observational.
• Frequentist sequential adjustments applied to Bayesian methods unnecessarily.
• Tool choice without methodology understanding; opaque outputs.
• No calibration check; predictions never compared to outcomes.
• Bayesian framing in stakeholder communication without explanation.
• Treating Bayesian as "always better" without considering use case.

Operating checklist

• Bayesian or frequentist choice documented per use case
• Priors documented and sensitivity-tested
• Bayesian sequential testing for experimentation
• MMM with experiment-informed Bayesian priors
• Decision thresholds documented (probability + lift)
• Credible intervals in reporting
• Decision-theoretic framing for stakeholders
• Calibration tracking (predictions vs outcomes)
• Tooling chosen and methodology understood
• Stakeholder education on Bayesian interpretation
• Documentation of prior choices and rationale
• Annual methodology review

---

Part of the Marketing Analytics series.