Free tool · no signup · runs in your browser

The Holdout Calculator.

Four numbers you already have — treated size, holdout size, two conversion rates. Out comes the truth: the real incremental lift, the incremental revenue, whether the result is statistically solid — and how much your dashboard is quietly over-claiming.

Try a typical shape:

—

Attributed conversions

—

Incremental (real) lift

—

Dashboard is over-claiming

—

—

All math runs in your browser — nothing is sent anywhere.

How to read this

The attributed number is what your dashboard already shows: how many people in the treated group converted. The incremental number subtracts the holdout’s baseline rate — the conversions that would have happened with no program at all. The gap between the two is what your dashboard is quietly claiming credit for.

The significance line runs a two-proportion z-test on your volumes. A big lift on a tiny holdout is a story, not a result — the test tells you which one you’re holding.

If the over-claim is small, the program is doing real work. If it’s large, you’re funding a channel that mostly rides the tide. Either way, now you know — instead of guessing.

What is a holdout group?

A holdout (control) group is the slice of your audience that deliberately gets nothing — no email, no push, no discount — while the rest of the segment gets the campaign. Their conversion rate is your baseline: what would have happened anyway. Every serious incrementality measurement in CRM and lifecycle marketing rests on it; without one, “attributed revenue” is a screenshot, not a fact.

  1. 01Split randomly before the send — same segment, same period, same measurement window. A holdout picked after the fact is a different population.
  2. 02Keep 5–10% of the segment in the holdout, and rarely go below ~1,000 people — smaller groups can’t detect the single-digit lifts most programs actually produce.
  3. 03Measure the same conversion for both groups, then read the difference, not the treated rate. That difference — and only that — is your program’s revenue.

The full standard behind this math — the Holdout Column → How I run measurement — Method →

Questions this tool gets asked

What is a holdout group in marketing?
The part of your audience that is deliberately excluded from a campaign so you can see what would have happened without it. Their conversion rate is the baseline; anything above it in the treated group is the campaign’s real (incremental) effect.
How big should a holdout group be?
5–10% of the segment is the usual working range, with a practical floor around 1,000 people. Below that, the math can’t reliably detect the 1–3 percentage-point lifts most lifecycle programs produce — this calculator’s significance line will show you exactly when that happens.
Why does my dashboard over-claim?
Because most attribution counts every converter who was touched, not every conversion that was caused. Customers who would have bought anyway get counted as campaign wins. Subtracting the holdout baseline removes exactly that inflation.
What is a good incremental lift?
Depends on the program: mature email promos often add 0.3–1 percentage point over baseline; well-built lifecycle journeys can add several. The honest answer is the one this calculator gives — a lift that survives the holdout subtraction and the significance test at your volumes.
Is my result statistically significant?
Enter your numbers above — the tool runs a two-proportion z-test and says so in plain words. Rule of thumb: small lifts need big holdouts; if p stays above 0.05, extend the window or enlarge the holdout before you present the number.

Want this run on your real stack, not a guess?

This is the arithmetic I run on every lifecycle program I own — with your data pulls, your holdouts, your P&L. Ex-Uber; every number on this site carries how it was measured.

Book a fit call or start with the €2,900 Diagnostic