Sample Ratio Mismatch (SRM)

Sample ratio mismatch (SRM) is when the actual traffic split in an A/B test significantly deviates from the intended ratio — you designed a 50/50 split but the data shows 52/48, or 53/47, by more than random chance would explain. It's one of the most important and under-checked experiment-validity signals, because a mismatched split is a red flag that SOMETHING IS BROKEN in the experiment (the randomization, the assignment, the logging, the data pipeline) — and when something is broken, the results are likely invalid, no matter how clear or significant the 'winner' looks. SRM is the silent invalidator: tests with SRM produce confident, compelling, and completely untrustworthy results.

Why a mismatched split signals a broken test and how to catch it: in a properly-functioning A/B test, the traffic split should match the intended ratio within the bounds of random chance (a 50/50 split won't be exactly 50.00/50.00, but it should be very close for large samples — random variation alone almost never produces a 52/48 split across hundreds of thousands of users), so a statistically-significant deviation from the intended ratio means the assignment isn't working as designed — and that means a bug, not a result. How to detect it: a chi-square goodness-of-fit test on the observed counts against the expected ratio (the standard SRM check — it tells you whether the observed split deviates from expected by more than chance, flagging SRM when the p-value is very low) — and the discipline is to run this check on EVERY test before trusting any result, because SRM is invisible to the naked eye (a 51/49 split looks fine but can be highly significant SRM at scale). Why SRM happens (the causes to investigate when it's flagged): broken or biased randomization (the assignment isn't truly random — a bug in the bucketing), redirect-and-loading issues (one variant loads slower or redirects, so users drop out differentially before being counted — a common cause, where the slower variant 'loses' users who never load it, biasing both the split AND the result), bot or duplicate filtering applied unevenly, logging and data-pipeline bugs (events from one variant under- or over-counted), and triggering or eligibility differences between variants. The critical point this entry must center: SRM doesn't just mean the split is off — it means the experiment is COMPROMISED, because whatever broke the split very likely also biased the outcome (the differential dropout that skewed the ratio also skewed who's in each group and thus the result), so a test with SRM should NOT be trusted or shipped regardless of how significant its result is — the result is an artifact of the bug, not a real effect, and the right response is to find and fix the cause and re-run, not to explain away the mismatch. The honest framing: sample ratio mismatch is the under-checked, silent invalidator of A/B tests — a significant deviation of the actual split from the intended ratio that signals a broken experiment whose results, however compelling, are likely artifacts of the bug rather than real effects; the discipline is running the SRM check (chi-square on observed counts) on every test before trusting it, treating any flagged SRM as an invalidating red flag (not a minor anomaly to explain away), investigating and fixing the cause (broken randomization, differential dropout, logging bugs), and re-running — because a test with SRM produces confident, compelling, and untrustworthy results, and catching it is the difference between shipping real wins and shipping artifacts.

Sample ratio mismatch matters in every A/B test and experimentation program as a critical, under-checked validity gate — because SRM silently invalidates tests, producing confident-but-untrustworthy results that get shipped as real wins when they're artifacts of a bug. It matters most in mature experimentation programs (where the volume of tests means undetected SRM ships bad decisions regularly) and any test whose result will drive a significant decision. The discipline is running the SRM check (a chi-square test on the observed counts against the intended ratio) on every test before trusting any result, treating any statistically-significant SRM as an invalidating red flag (the experiment is compromised, not a minor anomaly to explain away), investigating the cause (broken randomization, differential dropout from redirect/load issues, logging bugs), and fixing-and-re-running rather than shipping — because catching SRM is the difference between shipping real wins and shipping compelling artifacts of broken tests.

Sample ratio mismatch became a recognized experiment-validity gate as large-scale online experimentation matured (Microsoft, Google, and others documented it as a top cause of invalid tests); it remains under-checked despite being a simple chi-square test away, the silent invalidator that ships confident artifacts of broken experiments as real wins when the SRM check is skipped.

Etymology: source.

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Areas of marketing where sample ratio mismatch (srm) is a core concern: