SRM Checker

Q: What is Sample Ratio Mismatch (SRM)?

SRM is when the observed traffic split in an A/B test differs significantly from the split you configured (e.g. 52/48 on a 50/50 test). It signals a bug — a lossy redirect, uneven bot traffic, or a tracking/assignment error — that usually biases the result, so a failed SRM check should void the test.

Q: How is SRM detected?

With a chi-square goodness-of-fit test comparing observed counts to the counts expected under your intended split. For a two-variant test there is one degree of freedom, and a very small p-value means the mismatch is unlikely to be chance.

Q: What p-value threshold flags SRM?

Because the check runs on every test, use a strict threshold — commonly p < 0.001 (sometimes 0.0005). A looser threshold like 0.05 would raise false alarms constantly.

Q: What causes SRM?

Most often redirect-based split tests (users bounce during the extra hop on the redirected arm), uneven bot or crawler traffic, and consent or tracking gates that fire differently across variants. Trace assignment from randomization through delivery to logging.

Q: What should I do if SRM is detected?

Treat it as a hard stop: do not trust the result, investigate and fix the cause (redirects, bots, tracking, assignment), then re-run the test clean. A significant result on an SRM-failed test is not trustworthy.

Q: Does this tool send my data anywhere?

No. The chi-square calculation runs entirely in your browser; your counts are never transmitted or stored.

Before you trust any A/B test, check the split. If you configured 50/50 but the traffic came back 52/48 on real volume, that’s almost never chance — it’s a bug that probably biased your result. Enter your counts and get the chi-square SRM verdict instantly.

Sample Ratio Mismatch (SRM) is when an A/B test’s observed traffic split differs significantly from the split you configured — a red flag that a redirect, bot, or tracking bug corrupted the experiment. This checker runs a chi-square goodness-of-fit test on your counts and flags SRM at p < 0.001. A failed check should void the test: find and fix the cause before trusting any result. Runs in your browser.

The calculator

Variant A users (control) # iActual count assigned to A.

Variant B users # iActual count assigned to B.

Intended % to A % iYour configured split for A.

✓ No SRM — split looks valid

Sample Ratio Mismatch p-value

—

—chi-square

—observed split

—expected split

Export

Observed vs expected assignment
Variant	Observed	Expected	Difference

Walkthrough

How to use this tool

Enter the actual user counts per variant.Use the real number of users (or sessions — whatever you randomize on) assigned to each arm, from your experiment tool.
Enter the split you intended.The percentage you configured for the control — 50 for a 50/50 test, 90 for a 90/10 holdout, and so on.
Read the p-value.A chi-square goodness-of-fit test compares observed to expected assignment. A very small p-value means the split is unlikely to be random chance — a red flag.
Act on a failed check.If p is below ~0.001 you likely have SRM: investigate redirects, bot filtering, tracking, and assignment bugs — and don’t trust the test’s result until it’s fixed.
Export the check.Copy a share link, download the CSV, or print a one-page PDF for your QA record.

From the desk

RGM Expert Says

Real Growth Matters — Paid social practiceHow we use this tool with clients

Sample Ratio Mismatch is the validity check we run before reading any A/B result — because it’s the fastest way to catch a test that’s quietly broken. If you configured a 50/50 split but the data comes back 52/48 on serious traffic, that gap is almost never chance: it means a redirect is dropping users, a bot filter is hitting one arm harder, or assignment/tracking has a bug. And a broken assignment usually means the rest of the result is biased too.

The discipline that matters: a failed SRM check voids the test, full stop — you don’t get to keep the ‘win’ and explain the mismatch away. We treat it like a build failure: red means stop, find the cause, fix it, and re-run. The most common culprits are redirect-based split tests (latency drops users on the redirected arm) and bot/spam traffic landing unevenly.

Use a strict threshold. Because you run this check on every test, a loose p-value (like 0.05) would false-alarm constantly; the field standard is roughly p < 0.001 (often 0.0005). Below that, the mismatch is real enough to act on; above it, the split is consistent with random variation and you can trust the assignment.

The math

How it works

SRM is detected with a chi-square goodness-of-fit test comparing the observed counts to the counts you’d expect under your intended split:

Expected_i = Total × intended proportion_i

χ² = Σ (Observed_i − Expected_i)² ÷ Expected_i

For a two-variant test there is 1 degree of freedom, and the p-value is the probability of a χ² this large under a correct split:

p = erfc( √(χ² ÷ 2) )

Observed — the actual users assigned to each variant.
Expected — total users times your intended split.
Threshold — flag SRM at roughly p < 0.001, because the check runs on every test.

SRM detection via chi-square is the standard approach in the online-experimentation literature (Fabijan, Kohavi et al.). The calculation runs entirely in your browser. Companion to the statistics module.

Why it matters

The validity check that voids a ‘winning’ test

It’s tempting to skip SRM when a test shows a clean win — but that’s exactly when it matters most. A mismatch in the traffic split is a symptom that the experiment infrastructure misbehaved, and whatever broke the 50/50 (a lossy redirect, uneven bot traffic, a logging gap) almost certainly biased the metric you’re celebrating. A ‘significant’ result on an SRM-failed test is significant nonsense.

This is why mature programs automate SRM on every experiment and treat a failure as a hard stop. It’s a remarkably high-yield check: cheap to compute, unambiguous to act on, and it catches a class of bug that no amount of statistical sophistication at analysis time can repair. The split either matches what you configured or it doesn’t — and if it doesn’t, you have a bug, not a winner.

The fix is investigative, not statistical: trace assignment from randomization through delivery to logging. Redirect tests are the usual suspect (the redirected arm loses users who bounce during the extra hop); bot and crawler traffic landing unevenly is the second; and consent or tracking gates that fire differently across arms are the third. Resolve the cause, then re-run clean.

Benchmarks

How to read the result

Run this on every test, with a strict threshold.

Threshold convention from online-experimentation practice (commonly p < 0.001 or 0.0005).
p-value	Verdict	Action
≥ 0.01	No SRM	Split is consistent with chance; trust assignment
0.001 – 0.01	Suspicious	Watch closely; investigate if it persists
< 0.001	SRM detected	Void the test; find and fix the cause, then re-run

Voices worth trusting

What operators say

Getting numbers is easy; getting numbers you can trust is hard.

Ronny Kohavi

Trustworthy Online Controlled Experiments

A failed SRM check should void the experiment — the mismatch is a symptom that the same bug likely biased your metric.

RGM

Experimentation practice

Go deeper

Books worth reading

Trustworthy Online Controlled ExperimentsKohavi, Tang & XuThe definitive guide to A/B testing — and the source of SRM as a standard guardrail.Experimentation WorksStefan ThomkeWhy disciplined experiments, run trustworthily, beat intuition.Statistics Done WrongAlex ReinhartThe common statistical traps — validity checks like SRM exist to catch them.

Related on RGM

Keep learning

Course

Glossary

Concepts

Tools

FAQ

Common questions

What is Sample Ratio Mismatch (SRM)?