What is a holdout test?

An experiment withholding a marketing treatment from a randomized group — the outcome gap between treated and held-out measuring what the marketing actually caused, not what it touched.

What units can be held out?

Users for owned channels (email, CRM — enforced via suppression at the identity layer), geographies where user control is impossible (broad media), or platform audience splits with their caveats.

What do holdouts usually reveal?

Deadweight — retargeting, branded search, and discount programs harvesting conversions the held-out group completed anyway; attributed claims repricing to a fraction under the counterfactual.

Growth Marketing Glossary

Holdout Test

hold·out testnoun

Withhold it from some, and the gap tells you what it was worth — the design that prices marketing instead of crediting it.

Schematic — treated, held out, and the gap

Term: Holdout Test
Logic: Random holdout = the counterfactual
Units: Users, geos, or audience splits
Keeps finding: Deadweight - conversions that were coming anyway

Forms & parts of speech

holdout test · noun

The counterfactual by subtraction.

"The holdout test repriced the retargeting line in one quarter - 80% of its 'conversions' happened in the held-out group too."

Definition in plain terms

A holdout test deliberately withholds a marketing treatment from a randomly selected group — the holdout — while the rest receive it, so the difference in outcomes between the two measures what the marketing actually caused. It is INCREMENTALITY-TESTING's basic design and attribution's antidote: where ATTRIBUTION credits touchpoints for conversions that happened near them, the holdout asks the only question that prices spend — what happened where the marketing didn't go?

The mechanics

The design choices that decide validity: the unit of randomization (user-level holdouts for owned channels — email, push, CRM programs, where you control delivery person by person; GEO-HOLDOUT-TEST designs where user-level control is impossible — broad media, upper-funnel; audience-split designs inside ad platforms with their own caveats), the duration (long enough for the treatment's full effect window — a retention program's holdout read at two weeks measures the wrong thing), the contamination guards (the holdout must stay genuinely untreated: suppression lists enforced at the GOLDEN-RECORD identity layer, geo borders respecting media bleed, and no helpful CRM manager 'rescuing' held-out customers), and the power math (the CONVERSION-LIFT being hunted is often small relative to baseline noise — size the holdout via the same SAMPLE-SIZE discipline as any experiment, and resist the urge to shrink it because withholding feels expensive). What the design keeps finding, which is why it keeps being resisted: deadweight — branded-search clicks, retargeting impressions, and discount codes harvesting conversions that the held-out group completed anyway at nearly the same rate; the holdout converts 'this channel drove 10,000 conversions' into 'this channel CAUSED 1,200 of its 10,000 claims', a sentence attribution can't say. The cost framing that gets tests approved: the holdout's 'lost' revenue is the price of the only unbiased answer — small, bounded, and usually far cheaper than the spend the answer reprices.

When it matters

Holdout tests matter wherever spend is justified by correlated metrics — retargeting, branded search, CRM programs, loyalty perks, the channels whose attribution numbers are most flattered by selection — and at budget moments, where one quarter's holdout settles arguments years of dashboards prolonged. They matter most as a rhythm rather than an event: an always-on universal holdout (a small slice of every audience permanently untreated) reads program-level incrementality continuously, while one-off tests answer channel-level questions as they arise. The discipline is randomized units, enforced suppression, full-window reads, honest power - and the institutional courage to act on answers the dashboards spent years contradicting.

Worked example. A subscription retailer spends $640K/quarter on retargeting that attribution credits with a 6.2x ROAS, and the holdout settles the standing argument: 10% of the retargeting-eligible audience, randomly held out at the identity layer for one quarter, suppression verified weekly. The readout is the genre's classic: held-out users converted at 92% of the treated rate - the channel's true incremental contribution prices at 0.6x, a tenth of its attributed claim, because retargeting was mostly photographing people already walking toward the register. The reallocation keeps the defensible core (a lapsed-90-day segment where the holdout showed genuine 2.1x lift) and moves the rest into prospecting, where the next quarter's geo-holdout finds real incrementality at honest scale. The cultural change outlasts the test: a 5% universal holdout becomes permanent infrastructure, every channel's QBR now reports attributed AND incremental columns, and the phrase 'the holdout says' enters the company's vocabulary as the sentence that ends debates - the counterfactual, finally on the payroll.

Failure modes to watch. Holdouts contaminated by helpful hands - suppression unenforced, geo bleed ignored, held-out customers 'rescued' by CRM; reads taken before the effect window closes; holdouts sized by squeamishness instead of power math; one good test treated as permanent truth while creative, audiences, and baselines drift; and the institutional failure - answers obtained, dashboards obeyed anyway.

Synonyms & antonyms

Synonyms

holdout testholdout experimentcontrol-group test

Antonyms

attribution (the correlational rival)always-on holdout (the institutional version)

Origin & history

The holdout is experimental science's control group wearing marketing clothes - randomized controlled designs date to Fisher's 1920s agriculture, clinical trials standardized them, and digital marketing adopted the vocabulary as 'holdout' when identity systems made person-level withholding operationally possible; the deadweight findings have been embarrassing dashboards ever since.

Etymology: source.

Usage trends

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Common questions

What is a holdout test?: An experiment withholding a marketing treatment from a randomized group — the outcome gap between treated and held-out measuring what the marketing actually caused, not what it touched.
What units can be held out?: Users for owned channels (email, CRM — enforced via suppression at the identity layer), geographies where user control is impossible (broad media), or platform audience splits with their caveats.
What do holdouts usually reveal?: Deadweight — retargeting, branded search, and discount programs harvesting conversions the held-out group completed anyway; attributed claims repricing to a fraction under the counterfactual.

Related tools & calculators

toolCAC calculator
toolLTV:CAC calculator

Resources & people to follow

referenceWikipedia — treatment and control groups
referencePlatform conversion-lift methodology documentation
referenceRGM analysis — the holdout's lost revenue is the price of the only unbiased answer; make it a rhythm, not an event

Curated, non-competitor resources verified per term.

Related training

modulePerformance marketing

Disciplines

Areas of marketing where holdout test is a core concern:

Performance marketingGrowth strategy

Sources

trendsGoogle Trends — "incrementality test"