Email Holdout Lift Calculator
Your ESP reports the revenue it attributed to a send. This tells you how much you actually caused. Enter your treatment and held-out control groups and read the true incremental lift — and whether it is statistically real.
A holdout test withholds a campaign or flow from a randomly chosen control group, then compares revenue per recipient against the group that received it. The gap is the true incremental lift the email caused — the number attribution can’t give you. This calculator computes incremental revenue per recipient, total incremental revenue, the percentage lift, and a two-proportion z-test p-value so you know whether the lift is real or noise. Runs in your browser.
Email Holdout Lift Calculator inputs and result
First, a control group that is too small. Detecting a modest lift needs enough people in the held-out group to separate signal from noise; size the control to the smallest lift you would actually act on, not to whatever is convenient. Second, a window that is too short. If you measure the day after the send, you miss the delayed purchases that real buyers make a week or a month later, and you understate the effect. Match the window to the purchase cycle. Third, contaminating the control by sending it “something else” — a different campaign, a push notification, a retargeting ad you forgot was running. The moment the control receives any version of the treatment, the comparison stops being clean and the lift you read is no longer trustworthy. Keep the hold genuinely empty for the full window, randomize properly, and the number you get is the closest thing to causal truth a marketer can buy.
How to use this tool
- Run the send with a holdout.Randomly withhold a slice (often 5–10%) of the qualifying audience as a control group that receives nothing.
- Wait the full window.Let the measurement window run long enough to capture delayed purchases — days for an abandonment flow, up to ~90 days for a customer-level program.
- Enter both groups\u2019 numbers.Recipients, purchasers, and revenue for the treatment group and the held-out control.
- Read the incremental lift.The gap in revenue per recipient is the true lift the email caused; the p-value tells you whether it is real or noise.
- Export it.Copy a share link, download the CSV, or print a one-page PDF for the report.
RGM Expert Says
Your ESP will happily report “attributed” revenue for every send. The problem is that much of it would have happened anyway — a loyal customer who was going to reorder, then clicked your email first, gets counted as email revenue even though you didn’t cause the sale. A holdout is the only honest answer: withhold the send from a random control group and the gap between the two groups is the lift you actually created.
The number that reframes programs is the cannibalization check. Run this on a discount-heavy flow and watch what happens when the control group buys nearly as much as the discounted group — the percentage lift collapses and the p-value goes non-significant. That means the promotion mostly subsidized orders you already had. We’ve seen “hero” flows with huge attributed revenue post almost zero incremental lift once held out. Test your proudest, highest-attributed flow first — that’s exactly where the cannibalization hides.
Read both outputs together. A large per-recipient lift with a small p-value (under 0.05) is a real, scalable win. A large lift with a big p-value means your control was too small to be sure — widen it and re-run. And report the result in dollars per thousand recipients, not just a percentage; that’s the language a CFO actually trusts.
How it works
The calculator compares revenue per recipient between the two groups and tests whether the difference in purchase rate is statistically real:
- RPR lift % — the incremental RPR as a percentage of the control’s RPR.
- p-value — from a two-proportion z-test on the purchase rates; below 0.05 means the lift is unlikely to be chance.
- Total incremental — the per-recipient lift applied across everyone who received the send.
A two-sided two-proportion z-test; the control must be randomly assigned for the comparison to be valid. Runs entirely in your browser.
Why incrementality beats attributed revenue
Attribution answers “who clicked last before buying?” Incrementality answers the question that actually matters: “how many of these sales would not have happened without the email?” Those are very different numbers, and optimizing toward attributed revenue can push you to pour effort and discount margin into sales you were going to get for free.
A holdout makes the truth visible because random assignment makes the two groups identical in every way except one — whether they received the email. So any difference in their behavior is the email’s doing. There is no cleaner causal answer available to a marketer, and most brands never run one.
The discipline is to test your biggest, most-celebrated flows first, report lift in incremental dollars, and act on the result even when it stings. A program you can’t honestly measure is a program you can’t defend at budget time.
Attributed vs incremental: a worked example
Same flow, two ways of counting it.
| View | What it counts | Risk |
|---|---|---|
| Attributed revenue | Every sale with a prior click | Overstates |
| Incremental (holdout) | Only sales the email caused | Honest |
| Cannibalized | Discounted sales that would have happened anyway | Hidden cost |
What operators say
A holdout test answers the question attribution can’t: how much of this revenue would have happened anyway?
Apple’s Mail Privacy Protection has dramatically undermined the usefulness of opens, which were the dominant engagement signal.