The Holdout Group Is Now a Checkbox in Customer.io—Why Every Lifecycle Campaign Should Have One

In 1988, two ex-strategy consultants named Richard Fairbank and Nigel Morris convinced Signet Bank in Richmond, Virginia, to let them run its credit card division as a science experiment. At the time, almost every US credit card carried the same flat 19.8% APR and the same flat annual fee, regardless of who the customer was. Fairbank and Morris's bet was simple: a credit card company is actually an information business in disguise. Test enough offers against enough segments, with enough rigour, and you can price risk individually instead of charging everyone the same number.

So they ran thousands of test mailings against control groups...small slices of the audience deliberately held back to see whether the offer was the cause of the new account, or just along for the ride. The breakthrough product was the 1991 balance transfer offer, which only stood out as a winner because holdout-controlled testing surfaced it against the obvious-but-wrong alternatives. Signet spun the division out in 1994. It became Capital One.

Capital One built a multi-decade competitive advantage on a single discipline: comparing the people you marketed to against a held-back control group, every campaign, every time. That same discipline is now a checkbox inside Customer.io. The methodology that built a $50bn company is two clicks away. The only question is whether you'll use it.

Most lifecycle teams won't. They'll run A/B tests, call them holdouts, report attributed-revenue numbers up to the board, and then wonder why the forecasts keep missing. This post is about why those teams are wrong, why the operational excuse for skipping holdouts has been dead for years, and how to enable a holdout on every always-on campaign in your Customer.io workspace by next Monday.

A holdout is not an A/B test—and confusing them is the most common stats mistake on lifecycle teams

A/B tests answer "which variant is better?" Holdouts answer "should this campaign exist at all?" If your team uses the words interchangeably, your measurement is broken before you even start.

Here's the difference in one line. An A/B test assumes the campaign should run; it just optimises the version. A holdout interrogates the campaign itself—it compares getting the message against getting nothing.

The mental model is simple. Use A/B tests to improve what you've already decided to fund. Use holdouts to decide what to fund in the first place. The strongest measurement programmes layer both methods on a set cadence (Skai, 2025). If you want a deeper read on the variant-vs-variant side of this, the practical A/B testing guide covers what to test, sample sizes, and how to call a winner.

Where this mix-up shows up: the board deck. A 22% A/B winner over a 20% control variant is not a 22% lift over no-send. It's a 2-percentage-point improvement on top of a campaign whose true incremental contribution might be 5%, 10%, or even negative. If you've never run a holdout, you don't know which.

Why this matters now: attribution overstates real impact

Attributed conversions are correlation. Incremental lift is causation. They're not the same number, and the gap between them is bigger than most marketers realise.

Avinash Kaushik's foundational essay on the subject puts it bluntly: branded search is often 60–80% non-incremental, and retargeting is often 40–70% non-incremental. The same logic applies to lifecycle. A welcome series that "drives" 30% of new-user activation is often credited for behaviour that would have happened regardless of the email. So is the abandoned-cart sequence. So is most of what sits in your campaign list.

This is the same problem we covered in the dark funnel piece and in the broader lifecycle marketing reporting and attribution guide: last-click and last-touch models are flattering, and they're flattering by design. They tell you who got credit, not who caused the outcome. As we wrote previously, 56% of marketers struggle with attribution—and holdouts are the cheapest way to start telling the truth about it.

The checkbox: how to enable a holdout in Customer.io in two clicks

Customer.io's holdout test is set up from inside an A/B test. Two clicks: turn the message into an A/B test, then tick "Make this message a holdout test" on Variation B.

Here's the exact flow, per Customer.io's holdout test docs:

1. In your campaign workflow, click the message you want to test, then click "Turn into A/B test".
2. Set the traffic split. Variation A goes to your audience. Variation B is the holdout.
3. Click Edit Variation B and tick "Make this message a holdout test".

That's it. The held-out cohort gets routed to an internal message trap rather than to the user's inbox. Nothing is sent. But conversion goals keep firing against the held-out profiles, which is exactly what produces the lift comparison at the end.

This wasn't always two clicks. Customer.io's release note from 5 December 2022 spelled out the change: "Previously, we had a complicated recipe to perform holdout tests... now you can set up holdout tests with a simple checkbox." The old approach involved sending real messages to a black-hole address—a workable hack, but one that polluted your sending data and confused goal-tracking. The checkbox killed that. If your team's last memory of holdouts in Customer.io is the black-hole era, it's worth looking again.

For tests that need more than two variants, or holdouts that span an entire campaign rather than one message, use a Random Cohort branch instead. Drag it into the workflow and set the percentage flowing down each path. One of those paths can be an exit—an audience that enters the campaign, gets randomly assigned to the holdout cohort, and immediately leaves without receiving any messages.

A note for B2B teams: until late 2025, Customer.io's random cohorts assigned people to paths at the individual level. That broke for B2B, where you'd often want everyone in the same account to get the same experience so you weren't accidentally A/B testing within a single buying committee. The 6 November 2025 release fixed this by letting cohorts be assigned at the account, company, or other object level. If you're running account-level messaging with custom objects, this is the update that makes holdouts viable for you.

What the dashboard will and won't show you

Customer.io's A/B Test tab reports Chance To Beat Original (CTBO) across open rate, click rate, and conversion rate. For a holdout, only one of those columns matters.

Read the conversion column, not the opens. The held-out cohort has no opens by definition...the message didn't reach them. Opens and clicks on Variation A are interesting for diagnostics but they're not the comparison. The comparison is conversion rate on the cohort that got the message versus conversion rate on the cohort that didn't.

The numbers to report up the chain are:

• Absolute lift: treatment conversion rate minus holdout conversion rate, in percentage points
• Relative lift: that difference divided by the holdout conversion rate

If your treatment converted at 8% and your holdout converted at 6%, absolute lift is 2 percentage points and relative lift is 33%. Both numbers matter. Absolute lift tells you the size of the effect. Relative lift tells you the size of the effect relative to baseline behaviour.

Holdout sizing: rules of thumb that won't get you fired

Three rules of thumb for sizing the holdout cohort, depending on what kind of campaign you're testing.

5% for high-volume always-on campaigns. Welcome series, abandoned cart, post-purchase, milestone emails. These send constantly, so a 5% holdout still produces enough exposed-vs-held-out events to detect real lift within a 30–90 day window. The 5–10% range is the practitioner standard (Rejoiner, 2023).

10% for monthly newsletters and lower-volume sends. You only get 12 data points a year. The cohort needs to be bigger to reach statistical power within a realistic measurement window. Don't compensate by shortening the window—a noisy newsletter result over six weeks tells you nothing useful.

20–50% for new campaigns where you genuinely don't know if they hurt. Yes, that big. If you've just built a new lifecycle stream and you're not yet sure whether it helps, hurts, or does nothing, run a wide holdout. The downside of a negative-lift campaign running for a year before you noticed is bigger than the cost of a slow ramp. Once the campaign is established and you've measured a positive lift, you can shrink the holdout to 5%.

The measurement window: 30–90 days for revenue. Shorter for engagement signals like clicks. Pre-commit to the window before you flip the checkbox. No peeking at day 14 and panicking.

One trap to avoid: minimum detectable effect. If your campaign moves a 2% baseline by 0.2 percentage points, a 5% holdout might never produce a statistically significant result. Plan the holdout size against the lift you'd care about as a business, not the lift you hope for. If you'd cancel the campaign at sub-0.5pp lift, size for 0.5pp detection. If 0.1pp would be a win, you'll need a much bigger sample.

Klaviyo's parallel feature gives a sense of how seriously the industry takes sample-size pressure. Klaviyo's global holdout groups require a minimum of 400,000 profiles and recommend a three-month measurement window before you turn the feature on. Customer.io's per-campaign holdouts are more accessible to scale-ups, but the underlying maths is the same—you need enough events to see a real signal.

Holdout culture: what changes in your weekly review

Once every always-on campaign produces a lift number, your weekly review changes shape. The conversion rate is for optimisation. The incremental lift is for budget decisions. Those are two different conversations, and most teams have only ever had one of them.

The political problem is real and worth naming up front. Incremental lift is almost always lower than attributed conversion rate. Sometimes much lower. Someone on the team—usually whoever owns the campaign whose lift just got published—will hate the new number. That's not a problem to solve. That's the point of running the test.

Three practical changes to make in your reporting:

Add a lift column to your lifecycle marketing scorecard. Every always-on campaign gets a row. Conversion rate sits next to incremental lift. The board sees both. The team sees both. Nobody can pretend the higher number is the real one anymore.

Pre-commit to the measurement window in writing. Before the holdout starts, document the campaign name, start date, holdout percentage, measurement window, primary metric, and minimum detectable effect. Then write the decision rule: what lift result means continue, what means kill, what means redesign. When week three rolls around and the early numbers look bad, the document is what stops the team from killing the test early.

Treat a negative-lift result as a win, not a failure. A campaign that loses to no-send is the single highest-ROI finding a lifecycle team can produce. You save the send cost, you save the deliverability hit, you save the trust deposit with the inbox, and you free your team to spend that energy on something that actually moves the number. Killing a bad campaign is a better result than launching a new one.

Over a year, this compounds. Once every campaign reports a lift number, you can finally rank them honestly. The board deck stops being "we sent X emails and Y people converted" and starts being "here's the incremental revenue, by campaign, sorted." That's a different conversation about budget. That's a different conversation about headcount.

The campaigns where you should NOT hold out

Three categories of campaign should never have a holdout. Pre-commit to the list and put it in your campaign-governance doc.

Transactional and contractual. Order confirmations, shipping notifications, password resets, receipts, account-creation confirmations. Suppressing these creates a customer-service incident, not a measurement. There's no business question to answer—the campaign exists because the contract or the user expectation requires it. For what you can do with these sends as a side benefit, see our piece on turning transactional emails into a revenue channel without breaking the contractual job they're doing.

Regulatory. Renewal notices, terms-change notifications, breach disclosures, GDPR/CCPA opt-in confirmations, tax document availability emails. Anything you're legally required to send. The holdout question doesn't apply because the no-send branch isn't legal.

High-stakes one-offs where the cost of a wrong-sized holdout is bigger than the learning. Major product launches, crisis communications, founder letters at moments that matter, anything where reaching the full audience matters more than the experiment. These are also typically one-shot sends with no rerun, so the lift number wouldn't even be useful retrospectively.

Writing the list down matters more than what's on it. The point of governance is to make "yes, run a holdout" the default and "no, this one's exempt" the exception that needs a reason. Without a list, every campaign owner negotiates it case-by-case and reflexively opts out. With a list, the conversation is over before it starts.

Practical first step for next Monday

Pick your three highest-volume always-on campaigns. Enable a 5% holdout on each using the checkbox method. Document the holdout percentage, the start date, the measurement window (60 days is a reasonable starting point), the primary metric, and the decision rule. Set a calendar reminder.

When the reminder fires, calculate absolute lift and relative lift for each campaign. Bring those three numbers to your next weekly review. Refuse to use attributed conversions in the same conversation again.

You don't need to roll this out across every campaign on day one. Three is enough to establish the practice and the reporting cadence. Once those three are running and reporting, the fourth and fifth campaigns are easier conversations because the format is already in place. By the end of the quarter, you have a different relationship with your own data.

The hard part of holdouts has never been the methodology. It's been the operational lift of building them in tools that didn't support them natively. That excuse is gone. Customer.io's checkbox closes the gap. The companies that act on this in 2026 will be the ones still making confident forecasts in 2027, when their competitors are squinting at attributed conversion numbers and wondering why the year keeps missing plan.

Frequently asked questions

Q: How do I set up a holdout test in Customer.io?

In your campaign workflow, click the message you want to hold out, then click "Turn into A/B test." Set the traffic split between Variation A (the message) and Variation B (the holdout). Click Edit Variation B and tick "Make this message a holdout test." The held-out cohort gets routed to an internal message trap and never reaches the user, but conversion goals keep tracking. See Customer.io's holdout test docs for the full walkthrough.

Q: What's the difference between a holdout test and an A/B test in Customer.io?

An A/B test compares two versions of a message that both get sent. A holdout test compares sending a message against sending nothing at all. A/B tells you which variant wins. A holdout tells you whether the campaign should run in the first place. In Customer.io, a holdout is set up inside an A/B test by ticking "Make this message a holdout test" on one of the variations.

Q: What percentage of my audience should be in the holdout group?

5% for high-volume always-on campaigns is the practitioner standard. 10% for monthly newsletters and lower-volume sends, because you need more sample per measurement window. 20–50% for new campaigns where you're not yet sure if they help or hurt. Once you've measured a positive lift, you can shrink to 5%.

Q: How long should I run a holdout test for?

30 to 90 days for revenue and retention metrics. Shorter for engagement signals. The window should be long enough that customers who would have converted with the campaign have had time to do so, but short enough that you're not waiting a year for an answer. Pre-commit to the window before you start—peeking at day 14 and reacting to noise is the single biggest mistake teams make.

Q: Do conversion goals still track for the held-out cohort in Customer.io?

Yes, and that's the whole point. The held-out profiles never receive the message, but Customer.io continues to attribute conversion events to them. That's how the platform calculates the difference in conversion rate between the exposed cohort and the held-out cohort—the lift number.

Q: Should I run a holdout on transactional emails?

No. Transactional emails exist because the contract or the user expects them—suppressing an order confirmation or password reset creates a customer-service problem rather than a measurement. Regulatory emails fall in the same category. Pre-commit to a "no holdout" list as part of your campaign governance and stop relitigating it campaign by campaign.

Q: How do I calculate incremental lift from a Customer.io holdout test?

Absolute lift is treatment conversion rate minus holdout conversion rate, expressed in percentage points. Relative lift divides that difference by the holdout conversion rate. If your treatment converts at 8% and your holdout converts at 6%, absolute lift is 2pp and relative lift is 33%. Both belong in the report.

Q: Why is the incremental lift always lower than my attributed conversion rate?

Because attribution credits the email for conversions that would have happened anyway. The held-out cohort proves it—a portion of those customers convert even without receiving the message. Attributed conversion rate measures correlation, incremental lift measures causation, and the difference between the two is what the holdout exposes.

Q: When should I use a Random Cohort branch instead of the holdout checkbox?

Use the Random Cohort branch when you need more than two variants, when you want a holdout that spans an entire campaign rather than one message, or when you want a holdout cohort to exit the journey entirely. The checkbox method works inside a single A/B test on a single message. The Random Cohort branch works at the workflow level.

Q: Can I run a holdout on a whole campaign journey, not just one message?

Yes, using a Random Cohort branch. Drop the branch in at the start of the campaign workflow, set the percentage flowing down each path, and route the holdout cohort to an exit immediately. Profiles get assigned to the holdout, leave the campaign without receiving anything, and Customer.io tracks their conversions against the exposed cohort.

Q: How do I run a holdout test on a B2B audience without splitting decision-makers within the same account?

Use the November 2025 random-cohorts-by-object feature in Customer.io. It lets you assign the cohort at the account, company, or custom object level rather than the individual profile level. Everyone in the same account gets the same path—either all in the treatment or all in the holdout—so you're not A/B testing within a single buying committee.

Q: What's a minimum detectable effect and why does it matter for holdout sizing?

Minimum detectable effect is the smallest lift you could reliably measure with a given sample size and confidence level. If your campaign moves a 2% baseline by 0.2pp, a 5% holdout might not produce a statistically significant result. Size the holdout against the lift you'd act on as a business, not the lift you hope for. If you'd cancel below 0.5pp, plan to detect 0.5pp.

Q: How do I stop the team from killing a holdout test early because the numbers look bad?

Write the decision rule down before the test starts. Document the campaign, holdout percentage, start date, measurement window, primary metric, and what lift values would mean continue, kill, or redesign. When the early results look ugly at week two, the document is what stops the panic and protects the window.

Q: How do I report holdout results to a CFO who's used to attributed-revenue numbers?

Show both numbers in the same table, with the holdout-based incremental lift labelled as the causal measure and the attributed conversion as the correlational one. Explain that attribution credits the campaign for conversions that would have happened anyway, and the holdout strips those out. CFOs tend to recognise the underlying logic immediately because it's how they think about every other expense already.

Q: Is there a Customer.io minimum audience size for holdouts to work?

Customer.io doesn't enforce a minimum, but the maths does. Small audiences produce noisy lift numbers. As a working rule, you want at least a few hundred conversions in the held-out cohort over the measurement window for the result to be trustworthy. If your campaign sends to a few thousand people a month and converts at low single digits, plan for a longer window or a bigger holdout to compensate.

Sources

• Holdout tests — Customer.io Docs
• True holdout tests release note — Customer.io, 5 December 2022
• Random cohorts by account, company, or other objects — Customer.io, 6 November 2025
• Cohort tests — Customer.io Docs
• A/B tests — Customer.io Docs
• Marketing Analytics: Attribution Is Not Incrementality — Avinash Kaushik, Occam's Razor
• How to Measure The True Profitability of Your Email Campaign — Rejoiner, 2023
• Holdout Testing Marketing Guide — ATTN Agency, 2024
• Using Holdout Groups to Quantify Marketing Campaign Lift — Sunny Shapir, Bluecore Engineering, 2021
• Global Holdout Groups — Klaviyo, 2023
• Incrementality Testing vs A/B Testing — Skai, 2025
• Capital One: Leveraging Information-Based Marketing — Stanford Graduate School of Business
• Capital One — Wikipedia