What is sequential testing?

Experiment methods that allow continuous monitoring and valid early stopping by building the repeated looking into the statistics.

What problem does it solve?

Peeking — checking a fixed-horizon test repeatedly and stopping at significance inflates false positives; sequential methods correct for it.

What's the trade-off?

More complex math and conservative early-stopping boundaries (some lost power) in exchange for legitimate continuous monitoring.

Growth Marketing Glossary

Sequential Testing

se·quen·tial test·ing/sɪˈkwɛnʃəl ˈtɛstɪŋ/noun

Peeking at a normal test and stopping when it looks good inflates false positives — sequential methods make looking early legitimate.

Schematic — valid early stopping

Term: Sequential Testing
Solves: Peeking / continuous-monitoring false positives
Methods: Sequential probability ratio, alpha-spending, mSPRT
Benefit: Faster decisions without breaking validity

Forms & parts of speech

always-valid · adjective

Inference robust to continuous looks.

"The platform uses always-valid p-values — peeking won't lie to us anymore."

Definition in plain terms

Sequential testing is a family of experiment methods built for CONTINUOUS monitoring — they let you look at results as data arrives and stop early when the evidence is strong, WITHOUT the false-positive inflation that naive peeking causes in fixed-horizon tests. Where standard A/B testing requires committing to a sample size and not stopping early (because each peek is an extra chance to cross the threshold by luck), sequential methods build the repeated looking into the math, keeping the error rate honest.

The mechanics

The methods range from Wald's classic sequential probability ratio test to modern 'always-valid inference' (mixture SPRT, alpha-spending functions) used by experimentation platforms. The mechanism: instead of one significance threshold checked once, they define BOUNDARIES that account for the many looks — so a result that crosses the (appropriately stricter) boundary is genuinely significant despite the monitoring. The benefit is real efficiency: clear winners and losers can be called early (saving traffic), while inconclusive tests run on. The cost: the methods are more complex, and the early-stopping boundaries are conservative by design — you trade some power for the freedom to look.

When it matters

Sequential testing matters wherever continuous monitoring is desired (most modern experimentation), where stopping clear losers early protects revenue or users, and where the organizational reality is that people WILL peek (better to make peeking valid than to forbid it and watch it happen anyway). It's the principled answer to the peeking problem that p-value and sample-size discipline otherwise solve by prohibition. The honest framing: sequential methods let you look early legitimately — they don't let you skip having enough data for the effect you care about.

Worked example. A team keeps secretly peeking at fixed-horizon tests and shipping early 'winners' that evaporate — peeking has quietly turned their 5% false-positive rate into something far worse. Rather than fight human nature, they switch to a platform with always-valid sequential inference: now the dashboard can be watched daily, early stopping is legitimate, and clear losers get killed in days instead of weeks (protecting revenue). Replication rates recover because the early calls are now mathematically honest — the team got the monitoring they always wanted, with the validity they'd been quietly breaking.

Failure modes to watch. Applying sequential stopping rules to a fixed-horizon test (or vice versa); assuming sequential methods remove the need for adequate data; reading the conservative early boundaries as if they were the standard threshold; and 'going sequential' as a label without the actual corrected math.

Synonyms & antonyms

Synonyms

sequential testingsequential analysisalways-valid inference

Antonyms

fixed-horizon testingnaive peeking (the problem it solves)

Origin & history

Sequential analysis was developed by Abraham Wald during World War II (the sequential probability ratio test, published 1945-47) for efficient munitions quality control; modern 'always-valid' adaptations (mixture SPRT, alpha-spending) were brought to online experimentation by platforms like Optimizely in the mid-2010s to solve the peeking problem at scale.

Etymology: source.

Usage trends

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Common questions

What is sequential testing?: Experiment methods that allow continuous monitoring and valid early stopping by building the repeated looking into the statistics.
What problem does it solve?: Peeking — checking a fixed-horizon test repeatedly and stopping at significance inflates false positives; sequential methods correct for it.
What's the trade-off?: More complex math and conservative early-stopping boundaries (some lost power) in exchange for legitimate continuous monitoring.

Related tools & calculators

toolExperiment planner
toolFunnel drop-off analyzer

Resources & people to follow

referenceWald — Sequential Analysis (1947)
bookTrustworthy Online Controlled Experiments — Kohavi, Tang & Xu
referenceOptimizely / Statsig — always-valid inference docs

Curated, non-competitor resources verified per term.

Related training

moduleCRO & experimentation

Disciplines

Areas of marketing where sequential testing is a core concern:

ExperimentationAnalytics

Sources

trendsGoogle Trends — "sequential testing"