Impact Evaluation

Impact evaluation is the assessment of the causal effect of a program, policy, or intervention — what genuinely changed because of it, as distinct from what would have happened anyway. The central idea is the counterfactual: to know a program's true impact, you have to compare what happened with the program against what would have happened without it, and since you cannot observe both for the same group at the same time, impact evaluation uses comparison methods to estimate the missing counterfactual. That usually means some form of control or comparison group — people, regions, or units that did not receive the intervention but are otherwise similar — so the difference in outcomes can be attributed to the program rather than to other forces. Approaches range from randomized controlled trials, the gold standard for isolating cause, to quasi-experimental designs that construct a credible comparison when randomization is not possible. The goal throughout is causal: not just whether outcomes improved, but whether the program caused the improvement.

Impact evaluation matters because programs and campaigns are constantly credited with results they may not have caused. Outcomes change for many reasons — trends, seasonality, other initiatives, the broader environment — so the fact that things improved after an intervention does not mean the intervention caused the improvement. Without a counterfactual, organizations routinely overstate or misattribute impact, funding programs that did little and missing ones that did much. Impact evaluation guards against this by demanding evidence of causation, which leads to better decisions about what to continue, scale, or stop. In marketing, the same logic underlies incrementality measurement: the question of how many conversions a campaign actually caused, beyond those that would have occurred anyway, is an impact-evaluation question, and the holdout and control-group methods used to answer it are impact-evaluation methods applied to marketing spend.

The most important contrast is between impact evaluation and a simple before-and-after comparison. A before-and-after approach looks at outcomes before the program and after it, and attributes any change to the program. The problem is that many things change over time besides the intervention — trends, seasonality, the economy, other campaigns — so a before-and-after difference conflates the program's effect with everything else that was happening. If sales rose after a campaign, before-and-after credits the campaign, even if sales were already rising or a seasonal peak arrived. Impact evaluation avoids this trap by estimating the counterfactual: it compares the treated group not with its own past but with a comparison group that experienced the same trends and conditions without the program, so the difference isolates the program's causal effect rather than the backdrop it shared with everything else.

This is exactly why impact evaluation connects so directly to incrementality. Incrementality testing in marketing is impact evaluation applied to campaigns: it uses holdout or control groups that do not see the advertising to estimate what conversions would have happened anyway, and counts only the lift above that baseline as the campaign's true causal effect. Both reject the seductive but flawed before-and-after logic in favour of a counterfactual comparison. The difference between a believable impact estimate and a misleading one usually comes down to the quality of that comparison — how well the control group represents what would have happened without the program. A weak or biased comparison can be as misleading as no comparison at all, which is why good impact evaluation invests heavily in constructing a credible counterfactual through randomization or careful quasi-experimental design.

Using impact evaluation well means insisting on a counterfactual — comparing outcomes against what would have happened without the program, not just against the past — and constructing that comparison as credibly as the situation allows. Where possible, that means randomization, which gives the cleanest estimate of causal effect; where not, it means careful quasi-experimental design that builds a comparison group genuinely similar to the treated one. It means being honest about the quality of the comparison, since a biased control group undermines the whole estimate, and being clear about what is being measured — the causal effect of this program, over this period, on this outcome. In marketing, it means using incrementality methods like holdouts to measure what campaigns actually cause, rather than crediting them with conversions that would have happened anyway.

The failures are relying on before-and-after comparisons that conflate the program with every other change over time, using a weak or biased comparison group that does not represent the true counterfactual, attributing all observed change to the program when other forces were at work, and confusing correlation with causation. The discipline is to evaluate impact against a credible counterfactual — ideally through randomization, otherwise through careful comparison-group design — so that what gets credited to a program is what the program actually caused. Done this way, impact evaluation, and its marketing cousin incrementality testing, replace flattering but misleading attribution with honest evidence of effect.

Impact evaluation — assessing a program's causal effect against a counterfactual using comparison methods — isolates what an intervention actually caused, the same logic incrementality testing applies to marketing.

Etymology: source.

Search interest for this term over the last five years:

View interest-over-time on Google Trends →

Areas of marketing where impact evaluation is a core concern: