Experimentation & incrementality testing.

Marketing without experimentation is opinion-driven.

For most of the 20th century, marketing decisions were made by senior people with strong instincts in rooms with limited data. The good ones were right often enough to build careers; the rest were the source of the famous John Wanamaker line about half his advertising being wasted, except he didn't know which half. The arrival of digital and the analytics it produced was supposed to change that. In some places it has. In most, it has not — because the data exists but the discipline to make causal inferences from it doesn't.

Experimentation is that discipline. Every meaningful marketing change is treated as a hypothesis to be tested rather than a decision to be implemented. The test has a control group, a success metric defined before the test starts, a predetermined significance threshold, and a documented kill criterion. The output isn't an opinion about whether the change worked; it's a number with a confidence interval. The number determines what scales and what gets killed.

Where the methodology comes from.

The intellectual lineage of marketing experimentation runs through three distinct fields, each contributing a piece of the modern stack.

Agricultural statistics (1920s).

The mathematical foundations were laid by R. A. Fisher at Rothamsted Experimental Station in England, where he developed randomized controlled trials, the F-test, and the analysis of variance to answer questions about crop yields. His 1925 book Statistical Methods for Research Workers and 1935 follow-up The Design of Experiments are the source texts for nearly every concept in modern A/B testing — randomization, blocking, factorial designs, the p-value.¹

Clinical trials (1940s–1960s).

The British Medical Research Council's 1948 streptomycin trial — the first randomized controlled clinical trial in modern medicine — adapted Fisher's framework to drug efficacy testing. The discipline that emerged is the source of concepts marketers borrow today: control groups, blinding, intention-to-treat analysis, and the principle that the absence of statistical significance is not evidence of absence.²

Web experimentation (2000s).

The translation of clinical-trial discipline into digital marketing was led by two parallel communities. Ronny Kohavi, then at Amazon and later Microsoft, built one of the first industrial-scale A/B testing platforms and published widely-cited research on the practical pitfalls of online experimentation — including his canonical 2009 paper showing that even at Microsoft, the majority of confidently-shipped product changes had no measurable positive effect when tested rigorously.³ Kohavi's 2020 book Trustworthy Online Controlled Experiments, co-authored with Diane Tang and Ya Xu, is the working manual for most modern experimentation teams. In parallel, Sean Ellis, the marketer who coined the term "growth hacker" in a 2010 essay, brought experimentation discipline out of product teams and into marketing operations, establishing the cadence-driven testing rhythms now standard in growth marketing.⁴

Causal inference (2010s onward).

The most recent wave is the import of causal-inference methods from econometrics and epidemiology into marketing measurement. Judea Pearl's work on causal diagrams and the do-calculus, popularized for general audiences in The Book of Why (2018), and the synthetic-control method developed by Abadie, Diamond, and Hainmueller, are the theoretical bedrock under modern incrementality testing and modern marketing-mix modeling.⁵

How rigorous tests are structured.

There are five test types every growth program should know. They answer different questions, require different sample sizes, and have different failure modes.

Why most marketing tests are wrong.

Three failure modes account for the majority of misread test results, and they're worth knowing by name because they keep showing up in agency-produced "test results" reports.

Insufficient power.

A test's power is its probability of detecting a real effect of a given size, if one exists. Most marketing A/B tests run with too small a sample to detect lifts smaller than ~10–15%, which means the test will routinely return "no significant difference" even when a real 5% lift is present. Marketers then conclude "the change didn't work" when in fact the test was incapable of telling them. Power calculations should be run before the test starts; the answer often is "this test would need to run for 14 weeks to be powered, so we shouldn't run it."

Peeking.

Repeatedly checking a test in flight and stopping it when it looks significant is a near-universal practice that destroys statistical validity. The math is brutal: at standard p ≤ 0.05, a test that's peeked at daily over four weeks has a roughly 26% chance of producing a "significant" result purely by chance — over five times the 5% the threshold is meant to represent.⁶ Sequential analysis methods (always-valid p-values, Bayesian stopping rules) exist for tests that genuinely need to be monitored continuously, but most marketing teams just stop early without using them.

Selection effects and confounding.

The test wasn't really randomized. The treatment group accidentally got more weekend traffic, or higher-intent users, or saw the change during a promotional period the control didn't. These confounders are easy to introduce in marketing experiments where randomization happens at the ad-account level rather than the user level — and they almost always produce results that flatter whichever direction the analyst is hoping for.

Architecture is the prerequisite.

Experimentation as a service is largely a problem of having instrumentation honest enough to support it. Without clean data, no test result is trustworthy — and the most common pattern in growth programs we inherit is that they were running tests for years on data that wasn't fit for purpose.

The architectural prerequisites:

• Identity resolution — every meaningful test needs to track a user across sessions, devices, and platforms. Without identity resolution, "a 5% lift in conversion rate" is measuring something other than what you think.
• Server-side data collection — Apple's ITP, App Tracking Transparency, and the steady tightening of third-party cookie support have made browser-side data progressively less reliable since 2017. Server-side GTM, Tealium, or first-party endpoints reclaim the signal. More on server-side tracking →
• First-party data infrastructure — a clean event stream from your application, with user IDs, properties, and timestamps, into a warehouse you control. The platforms' own SDKs and tags are not a substitute. They report on themselves.
• Holdout discipline — the operational habit of always keeping a small percentage of users in a true holdout, untouched by marketing, to serve as the natural control for any incrementality question.
• An honest dashboard — daily metrics that the people making decisions actually look at, built on the warehouse data rather than the platforms' self-reporting. More on the analytics layer →

It's a culture, not a tool.

The hardest part of experimentation isn't the statistics. It's the willingness to ship a test, see it return a negative result, and act on the negative result — kill the change, reallocate the budget, walk back the campaign that someone senior was attached to. This is a cultural commitment that most organizations talk about and very few actually have.

The good experimentation cultures share three traits. First, they treat negative results as the same kind of asset as positive ones — every test answers a question, and "no, this didn't work" is a real answer worth documenting. Second, the senior team practices what they preach: when an executive's pet idea tests negative, it gets killed, publicly, and the team trusts the system more for having seen it happen. Third, the experimentation backlog is written down and visible. What gets prioritized for testing tells you what the org actually thinks is its biggest unknown.

Common questions.