Bandits & Bayesian testing
Bandits aren’t a better A/B test — they’re a different objective. This module covers when to minimize regret with a bandit versus learn cleanly with an A/B test, the explore-exploit tradeoff and the core algorithms (epsilon-greedy, UCB, Thompson sampling), Bayesian A/B testing and expected loss, and contextual bandits for personalization — with a clear rule for choosing by the decision’s shelf life.
What you will learn
When bandits beat A/B (and when they don’t)
A multi-armed bandit dynamically shifts traffic toward better-performing variants during the test, instead of splitting evenly and deciding at the end like an A/B test. Use a bandit when your goal is to minimize regret — capture as much value as possible while learning, on short-lived or high-velocity decisions (headlines, campaign creative, time-boxed promotions). Use a clean A/B test when your goal is to learn — get a clear, defensible causal read you can generalize and bank. The right tool depends on whether you most need the answer or the earnings.
RGM tool · Get decision-ready Bayesian numbers — probability B beats A, expected uplift, and expected loss — with the Bayesian A/B Test Calculator.
The trap is treating bandits as a strictly-better A/B test because they sound more sophisticated. They’re not better — they’re different. Bandits sacrifice clean statistical inference (clear significance, a stable effect size you can quote forever) in exchange for earning more during the experiment. For a foundational learning you’ll reuse for years, that trade is bad; for ‘which of five subject lines wins this one send,’ it’s exactly right.
People pick bandit vs A/B by which sounds smarter. The real question is simpler: how long does this decision matter, and what does it cost to be a little wrong while you learn?
If the decision is long-lived and you’ll generalize from it (a checkout flow, a pricing model), I want a clean A/B — the durable, defensible learning is worth the regret of splitting traffic evenly. If it’s ephemeral and the only goal is to win this instance (a campaign, a flash-sale banner), I want a bandit minimizing regret.
Match the method to the decision’s shelf life, not to which one impresses the room.
A bandit isn’t a better A/B test — it’s a different objective. One earns while it learns; the other learns cleanly. Pick by what the decision needs.
Explore vs exploit
Every bandit balances exploration (sending traffic to uncertain variants to learn which is best) against exploitation (sending traffic to the current best to earn). An A/B test is pure exploration until it ends, then pure exploitation; a bandit blends them continuously, exploring less as it grows confident. This explore-exploit tradeoff is the whole idea: spend just enough traffic learning to stop spending on losers, while cashing in on winners as soon as you reasonably can.
Claim: Multi-armed bandits formalize the explore-exploit tradeoff — allocating traffic to maximize cumulative reward (minimize regret) rather than to produce a clean end-of-test comparison. Source: Standard reinforcement-learning / bandit theory. Context: Bandits optimize earnings while learning; A/B tests optimize the clarity of the final learning. Different objectives, not better/worse.
The main algorithms
Three algorithms cover most uses. Epsilon-greedy — exploit the best arm most of the time, explore randomly a fixed fraction (epsilon); simple but crude. Upper Confidence Bound (UCB) — favor arms by their optimistic potential, exploring those with high uncertainty; principled and deterministic. Thompson sampling — sample each arm’s value from its probability distribution and play the winner; elegantly balances explore/exploit and is the most widely used in practice. You rarely implement these by hand — but knowing them tells you what your tool is doing.
Play the current best arm a fraction (1−ε) of the time, pick randomly the rest. Dead simple, but it explores blindly and keeps wasting a fixed slice on known losers.
Rank arms by an optimistic estimate (value plus an uncertainty bonus), so under-explored arms get a look. Deterministic and well-grounded in theory.
Draw a sample from each arm’s posterior distribution of its true rate and play whichever sample is highest. Naturally explores uncertain arms and exploits confident ones; the de facto standard.
Bayesian A/B testing
Bayesian A/B testing (distinct from a bandit) keeps the even split of a normal test but reports probabilities instead of p-values: ‘there’s a 95% probability B beats A, with an expected lift of 4% and an expected loss of 0.1% if you’re wrong.’ That ‘expected loss’ gives a natural, business-friendly stopping rule — ship when the risk of being wrong is acceptably small. It shares math with Thompson sampling (both reason about posterior distributions), which is why Bayesian tools and bandits often come from the same toolkit.
Why teams like it: ‘92% chance to win, expected 4% lift’ is a sentence an executive can act on, where ‘p=0.04’ is not. But two cautions. It is not immune to peeking abuse — naive ‘stop when probability-to-beat crosses 95%’ can still inflate error if you ignore expected loss and minimum sample. And it requires a prior; with sensible defaults this is fine, but understand your tool’s assumptions rather than trusting the friendly number blindly.
Contextual bandits for personalization
A contextual bandit uses information about each user (context — device, source, geo, behavior) to choose the best variant for that user, rather than one global winner. It’s the bridge from A/B testing to personalization: instead of ‘which headline wins overall,’ it learns ‘which headline wins for mobile paid-social visitors vs returning desktop users.’ Powerful for recommendations, content, and offers — and considerably more complex to build, validate, and debug, so adopt it when personalization value clearly justifies the engineering.
Contextual bandits are seductive: ‘the algorithm will find the best experience for everyone automatically.’ Then teams discover they’ve built a black box they can’t debug, optimizing toward a metric on segments too small to trust.
My rule: don’t reach for contextual personalization until a plain A/B test shows the variants genuinely differ by segment — that segment-level interaction is the evidence that personalization has something to find. No interaction, no reason to personalize; you’d just add complexity for a single global winner a simple test already gave you.
Personalization is earned with evidence of heterogeneity, not assumed because it sounds advanced.
Bandits sound advanced, so teams reach for them by default and quietly forfeit clean learning. The result is a program that optimizes lots of small things and understands none of them.
I flip the default: A/B is the assumed method, and choosing a bandit requires a one-line justification — ‘this decision is short-lived and we only care about winning this instance, not learning a reusable lesson.’ If you can’t write that sentence honestly, you want an A/B.
Defaulting to the method that produces durable knowledge, and treating regret-minimization as the exception you argue for, keeps the program learning instead of just twitching.
Where each fits
A quick map. Clean A/B / Bayesian A/B: foundational, long-lived decisions you’ll generalize (checkout flow, pricing, core page templates) — you need a defensible, reusable learning. Bandit: short-lived, high-velocity, exploit-now decisions (campaign creative, email subject lines, news headlines, time-boxed promos) where minimizing regret beats clean inference. Contextual bandit: ongoing personalization where the best experience genuinely differs by user and the value justifies the complexity. Match method to objective and shelf life.
Implementation and tooling
Most teams don’t hand-build bandits — major experimentation and personalization platforms offer ‘auto-allocate’ or ‘multi-armed bandit’ modes (typically Thompson-sampling-based) and Bayesian reporting out of the box. The work is configuring them correctly: choosing the reward metric carefully (bandits optimize exactly what you tell them to, including the wrong thing), setting minimum exploration so you don’t lock in early noise, and monitoring for the same validity issues (SRM, tracking) any experiment needs. The algorithm is the easy part; the metric and the guardrails are where it goes right or wrong.
- Is a multi-armed bandit always better than an A/B test?
- No. A bandit minimizes regret (earns more while learning) but sacrifices clean statistical inference. For a foundational learning you’ll reuse, a clean A/B is better; for short-lived, exploit-now decisions, a bandit is better. Match the method to the goal.
- What is Thompson sampling?
- A bandit algorithm that samples each arm’s value from its posterior distribution and plays the highest sample, naturally balancing exploration and exploitation. It’s the de facto standard behind most ‘auto-allocate’ features and is closely related to Bayesian A/B testing.
- Does Bayesian testing stop the peeking problem?
- Not automatically. Bayesian outputs are more intuitive and support expected-loss stopping rules, but naively stopping the moment probability-to-beat crosses a threshold can still inflate error. Respect minimum sample and expected-loss thresholds, and understand the prior your tool uses.
Where bandits go wrong
Bandit and Bayesian methods fail when: a bandit is used for a decision that needed a clean learning, the reward metric is mis-specified (so it optimizes the wrong thing), exploration is too low and early noise gets locked in, Bayesian results are over-trusted without understanding priors or expected loss, contextual bandits are deployed without evidence of segment differences, and validity checks (SRM, tracking) are skipped because ‘the algorithm handles it.’
Auto-allocating a foundational decision robs you of a clean, reusable causal read.
A bandit relentlessly optimizes exactly the metric you give it, including a flawed one.
Cutting exploration too soon can lock in an arm that was just early-lucky.
‘95% to win’ misused as an instant stopping rule still inflates error.
Contextual bandits on segments with no real difference add complexity for nothing.
Your bandit/Bayesian checklist
Advanced methods reward matching tool to objective. Tick what is genuinely true.
Prove it. Earn your passcode.
Ten questions, CASE method (Context · Analysis · Strategy · Execution). Pass at 90% to unlock this module’s completion passcode — retake as many times as you like.