Bayesian A/B Test Calculator

Q: What is Bayesian A/B testing?

An approach that reports the probability one variant beats another (and by how much), plus the expected loss of a wrong decision, instead of a p-value. It answers the questions teams actually have — how likely is B better, and what do I risk — in plain, decision-ready terms.

Q: How is this different from a p-value / significance test?

A p-value is the probability of the data if there were no effect; it is not the probability your variant is better. Bayesian testing gives that direct probability plus an expected-loss risk measure, which is why many modern testing tools default to it.

Q: What is expected loss and why does it matter?

The average downside (in conversion rate) if you pick the variant and the control was actually better. It’s the real ship signal: when expected loss is negligible you can ship even below 95% probability, because being wrong costs almost nothing.

Q: When should I ship the variant?

When P(B>A) is high (commonly ≥95%) AND expected loss is negligible. The two together calibrate the decision better than a binary significant/not. Keep testing if either is short, and respect a minimum sample so you don’t stop on noise.

“p = 0.04” isn’t a decision. “96% chance the variant wins, +8% uplift, and a rounding-error downside if we’re wrong” is. Enter your conversions and get the probability B beats A, the expected uplift, and the expected loss — the numbers teams actually act on.

Bayesian A/B testing reports the probability the variant beats the control and the expected loss if you ship it and you’re wrong — not a p-value. This calculator models each variant’s rate as a Beta posterior and runs ~40,000 Monte Carlo samples to give P(B>A), expected uplift, and expected loss. Ship when the probability is high and the expected loss is negligible. Runs in your browser; it complements, not replaces, sample-size discipline and validity checks.

The calculator

Control visitors # iUsers in A.

Control conversions # iConversions in A.

Variant visitors # iUsers in B.

Variant conversions # iConversions in B.

—

Probability the variant beats control

—

—expected uplift

—expected loss if wrong

—A vs B rate

Export

Conversion rates compared
Variant	Visitors	Conversions	Rate

Walkthrough

How to use this tool

Enter visitors and conversions for each variant.Control (A) and variant (B) — the totals at the point you’re analyzing.
Read the probability B beats A.Unlike a p-value, this is the direct, intuitive answer: the chance the variant is genuinely better, given the data.
Check the expected loss.The expected downside (in conversion rate) if you pick B and you’re wrong. A tiny expected loss is the real signal to ship — not the probability alone.
Decide with both numbers.Ship when P(B>A) is high AND expected loss is negligible. Keep testing when either is short — and respect a minimum sample so you don’t stop on noise.
Export the result.Copy a share link, download the CSV, or print a one-page PDF for your test record.

From the desk

RGM Expert Says

Real Growth Matters — Paid social practiceHow we use this tool with clients

We reach for the Bayesian view when stakeholders need a decision, not a statistics lecture. ‘There’s a 96% probability the variant is better, with about 8% expected uplift and a 0.05% downside if we’re wrong’ is a sentence a CMO can act on; ‘p = 0.04’ is not. The two outputs that actually drive the call are the probability to beat and the expected loss.

Expected loss is the underused half. It answers the question that matters when you ship: if I’m wrong and the control was actually better, how much do I give up? When that number is negligible, shipping the variant is safe even at a probability below the classic 95% — because the cost of being wrong is tiny. That risk framing is more decision-useful than a significance gate.

Two honest cautions. Bayesian results are not immune to peeking abuse — naively stopping the instant the probability crosses 95% can still inflate error, so respect a minimum sample and let expected loss, not just probability, gate the decision. And this uses an uninformative prior (Beta 1,1); if you have strong prior data, a calibrated prior changes the numbers. Use it to make the call legible, not to skip the discipline.

The math

How it works

Each variant’s true conversion rate is modeled as a Beta distribution updated by the data (a uniform Beta(1,1) prior plus observed conversions and non-conversions):

rate ~ Beta(conversions + 1, visitors − conversions + 1)

The tool draws tens of thousands of samples from each variant’s distribution and compares them to estimate the decision metrics:

P(B > A) = share of samples where B’s rate exceeds A’s

Expected loss = average of max(rate_A − rate_B, 0) across samples

P(B>A) — the probability the variant is genuinely better.
Expected uplift — the average relative improvement of B over A across samples.
Expected loss — the average downside if you pick B and A was actually better; your risk.

Monte Carlo over beta-binomial posteriors (uniform prior), ~40,000 samples, computed in your browser; numbers vary slightly run to run. For the frequentist view use the significance calculator; size tests with the sample-size calculator.

Why it matters

Probability and expected loss beat a lone p-value

A p-value answers a question almost nobody actually asked: ‘how surprising is this data if there were no effect?’ The Bayesian outputs answer the questions teams really have: how likely is B better, and what do I risk by shipping it? That reframing — from a significance gate to a probability plus a risk — is why many modern testing tools default to Bayesian reporting.

Expected loss is the quietly powerful part. It lets you ship a variant that’s ‘only’ 92% likely to win when the downside of being wrong is a rounding error, and it stops you shipping an 96%-likely winner when the potential loss is still material. Decisions made on probability-plus-risk are simply better calibrated than decisions made on a binary ‘significant or not.’

None of this repeals the discipline. Bayesian methods still need enough data, still get abused by peeking if you stop the instant a threshold is crossed, and depend on the prior. Use this calculator to make the decision legible and risk-aware — alongside a pre-set minimum sample and the validity checks (like SRM) every trustworthy test needs.

Benchmarks

Reading the result

Decide on probability AND expected loss together — not probability alone.

Guideline thresholds; calibrate to your risk tolerance and the cost of a wrong decision.
P(B>A)	Expected loss	Decision
≥ 95%	negligible	Ship the variant
90–95%	tiny	Likely ship — the downside is trivial
< 90%	any	Keep testing (respect minimum sample)

Voices worth trusting

What operators say

Getting numbers is easy; getting numbers you can trust is hard.

Ronny Kohavi

Trustworthy Online Controlled Experiments

Ship on probability AND expected loss — a 92%-likely winner with negligible downside beats a 96% one whose potential loss is still material.

RGM

Experimentation practice

Go deeper

Books worth reading

Trustworthy Online Controlled ExperimentsKohavi, Tang & XuFrequentist and Bayesian methods, and the discipline behind both.Bayesian Methods for HackersCameron Davidson-PilonIntuitive, computational Bayesian inference for practitioners.Statistics Done WrongAlex ReinhartThe traps — including misreading p-values — that Bayesian framing helps avoid.

Related on RGM

Keep learning

Course

Glossary

Tools

Services

FAQ

Common questions

What is Bayesian A/B testing?

An approach that reports the probability one variant beats another (and by how much), plus the expected loss of a wrong decision, instead of a p-value. It answers the questions teams actually have — how likely is B better, and what do I risk — in plain, decision-ready terms.

How is this different from a p-value / significance test?

A p-value is the probability of the data if there were no effect; it is not the probability your variant is better. Bayesian testing gives that direct probability plus an expected-loss risk measure, which is why many modern testing tools default to it.

What is expected loss and why does it matter?

The average downside (in conversion rate) if you pick the variant and the control was actually better. It’s the real ship signal: when expected loss is negligible you can ship even below 95% probability, because being wrong costs almost nothing.

When should I ship the variant?

When P(B>A) is high (commonly ≥95%) AND expected loss is negligible. The two together calibrate the decision better than a binary significant/not. Keep testing if either is short, and respect a minimum sample so you don’t stop on noise.

Does Bayesian testing avoid the peeking problem?

Not automatically. Naively stopping the instant probability crosses a threshold can still inflate error. Respect a pre-set minimum sample, let expected loss gate the decision, and run validity checks like SRM.

How does the calculator compute the result?

It models each variant’s rate as a Beta posterior (uniform prior plus your data) and draws ~40,000 Monte Carlo samples to estimate P(B>A), expected uplift, and expected loss. It runs entirely in your browser; numbers vary slightly run to run.

Related tools