The Practical A/B Testing Guide for Email Marketers

In May 1747, a naval surgeon named James Lind was sailing aboard HMS Salisbury in the English Channel when he decided to run an experiment that would change medicine forever.

Scurvy was killing British sailors at a shocking rate. Theories about the cause were everywhere — bad air, poor constitution, salt water. Everyone had an opinion. Nobody had proof.

Lind selected 12 sailors with similar scurvy symptoms, put them all in the same quarters with the same base diet, and gave each pair a different treatment: cider, dilute sulphuric acid, vinegar, seawater, a medicinal paste — and two oranges with one lemon. After six days, the citrus pair recovered. The others didn't.

Widely recognised as the first controlled clinical trial in history, Lind's experiment worked because he did one thing right: he held everything constant except the single variable he was testing.

Here's the tragic part. Lind didn't clearly act on his own results. The British Navy waited 48 years before mandating lemon juice for sailors. Tens of thousands of lives lost — not because the data was wrong, but because nobody used it.

That's the A/B testing problem in a nutshell. Most email marketers either don't test at all, or they run tests and don't act on the results. Only 59% of companies A/B test their email campaigns. The rest are guessing.

This guide is for the other 41% — and for the 59% who test but want to do it properly.

---

What Is A/B Testing in Email Marketing?

A/B testing (also called split testing) is the practice of sending two versions of an email to different segments of your audience and measuring which performs better. Version A goes to one group, Version B to another. You track a metric — open rate, click rate, conversion — and the winner becomes your new standard.

The entire point is to replace opinion with evidence. You're not guessing what subject line works better. You're knowing.

Done well, A/B testing compounds. Every winning test raises your baseline. Run enough tests over a year and you've built an email programme that consistently outperforms anything you could have designed by instinct alone.

---

Why Should You Bother A/B Testing Emails?

Short answer: the numbers are too good to ignore.

Marketers who A/B test emails achieve 83% higher ROI than those who don't. Subject line testing alone can lift open rates by up to 49%. And the Notion team — using Customer.io's A/B testing tools — achieved a 20% open rate lift by repositioning a single word in a subject line.

That's not a redesign. Not a new template. Not a bigger budget. One word.

Jonny Bigelow, Growth Marketing Lifecycle at Notion, put it plainly: "A/B testing is a cornerstone of our lifecycle marketing function at Notion. We have used it to power product launches to maximize impact."

The Notion team ran hundreds of experiments through Customer.io, driving millions in revenue and achieving 49–51% open rates on personalised onboarding sequences. It didn't happen by luck.

---

What Should You A/B Test in Emails?

You can test almost anything in an email. The question is what to test first, and what to test systematically. Here's a priority framework:

1. Subject Lines

Subject lines are the highest-leverage test you can run because they affect everyone. A 5% open rate improvement means 5% more of your list actually reads anything you wrote.

What to test:

• Length (short vs. long)
• Personalisation (with vs. without first name)
• Questions vs. statements
• Emojis vs. no emojis
• Urgency or scarcity vs. curiosity
• Direct ("Your invoice is ready") vs. intriguing ("Something's waiting for you")
• Word choice — as Notion proved, repositioning a single word drove a 20% lift

Important caveat: Since Apple's Mail Privacy Protection (MPP) launched in 2021, open rate data is inflated for Apple Mail users (roughly 46% of all email clients, according to Litmus). Use open rate as a directional signal for subject line tests, but validate against click-to-open rate (CTOR) for a cleaner read.

2. Send Times and Days

When you send matters — but the "best time to send" varies by audience. The published averages (Tuesday at 10am, etc.) are aggregates across millions of sends. Your audience may behave completely differently.

What to test:

• Morning vs. afternoon sends (try 8–10am vs. 1–3pm)
• Weekday vs. weekend (especially for B2C audiences)
• Day of week (Tuesday vs. Thursday for B2B is a common starting point)
• Send time optimisation features — Customer.io's intelligent delivery sends each user a message when they're most likely to engage based on their individual history

31% of email marketers plan to add send time optimisation to their strategy in the next cycle. If your competitors aren't doing it yet, this is a quick win.

3. Email Content and Copy

Once someone opens your email, the content determines whether they act. Content tests take longer to set up but often produce the biggest revenue impact.

What to test:

• Opening line (question vs. statement, problem vs. benefit)
• Email length (short punchy vs. long detailed)
• Plain text vs. HTML design
• Single-column vs. multi-column layout
• Images vs. no images
• Video thumbnail vs. static image
• Tone (formal vs. casual)
• Personalisation depth (first name only vs. personalised recommendations)

4. Calls to Action (CTAs)

The CTA is where intent becomes action. Small changes here directly affect conversion rate.

What to test:

• Button text ("Get started" vs. "Start your free trial" vs. "Try it free")
• Button colour
• Button placement (above the fold vs. below)
• Single CTA vs. multiple CTAs
• Text link vs. button
• Urgency framing ("Claim your spot" vs. "Register now")

Emails with a single CTA can increase clicks by up to 371% compared to emails with multiple CTAs. Personalised CTAs convert 202% better than generic ones.

5. From Name and Sender Address

The sender line is read before the subject line. It's one of the most underrated test variables in email marketing.

What to test:

• Company name only ("NerveCentral") vs. person name ("Alex from NerveCentral")
• Named founder vs. generic team address
• Branded subdomain vs. main domain

---

What Is Statistical Significance and Why Does It Matter?

Statistical significance is the measure of whether your test result reflects a real difference — or just random noise.

Here's the core idea: if you send Version A to 100 people and Version B to 100 people, and A gets 22 opens and B gets 25 opens, is B actually better? Maybe. Or maybe that 3-person difference is random chance. With 100 people per variant, you can't tell.

Statistical significance answers the question: how confident are you that what you observed is real?

What confidence level should you use?

95% confidence is the email marketing industry standard. This means there's only a 5% chance the result you're seeing is due to random variation. Some teams use 99% for high-stakes decisions, but 95% is the baseline that makes you a marketer rather than a gambler.

How does Customer.io calculate this?

Customer.io calculates and displays statistical significance in real time in your A/B test dashboard. When your test hits significance, you'll see it flagged — you don't need to run the maths yourself. But understanding why statistical significance matters helps you set up your tests properly from the start.

How much does sample size matter?

A lot. This is the most common mistake in email A/B testing.

The rule of thumb: aim for at least 1,000 recipients per variation as an absolute floor, and ideally 5,000+ per variation for reliable results. If you're testing something with a low baseline rate (like conversion from email to purchase), you need even larger samples.

Here's a quick reference:

Baseline Rate	Minimum Detectable Effect	Sample per Variation
20% open rate	Detect a 5pp lift (to 25%)	~2,400
20% open rate	Detect a 2pp lift (to 22%)	~15,000
3% click rate	Detect a 1pp lift (to 4%)	~6,500
1% conversion	Detect a 0.5pp lift (to 1.5%)	~25,000

Small lists (under 2,000 total) should focus on the highest-impact variables (subject lines, CTAs) and accept wider confidence intervals, or run longer-horizon tests across multiple sends.

Use Optimizely's sample size calculator or Evan Miller's calculator for precise numbers.

---

What Are the Most Common A/B Testing Mistakes?

Knowing what to avoid saves you from wasting months on inconclusive data.

Mistake 1: Ending Tests Too Early

The "peeking problem" — checking results before statistical significance is reached and calling a winner based on early data — is the most damaging testing mistake. Early results are noisy. What looks like a 15% lift at hour three might disappear by hour 48. Set a stopping criterion before you start, and stick to it.

For email: wait 24–72 hours after send. 85% of responses arrive in the first 24 hours, but B2B audiences often respond more slowly. For drip campaigns with ongoing enrolment, run tests for full-week increments to capture day-of-week variation.

Mistake 2: Testing Multiple Variables at Once

If you change the subject line and the CTA and the send time, you won't know which change caused the lift. Always isolate one variable. Multi-variate testing (testing multiple variables simultaneously) is possible, but it requires exponentially larger sample sizes and more sophisticated analysis than most email teams have capacity for.

Mistake 3: Using Open Rate as Your Only Metric

Post-MPP, open rates are noisy. Apple Mail pre-loads tracking pixels for privacy-focused users, artificially inflating opens by up to 18 percentage points. Use open rate directionally for subject line tests, but validate with CTOR (click-to-open rate) and conversion rate wherever possible.

Mistake 4: Not Defining a Winning Metric Upfront

52.8% of CRO professionals admit they have no standardised stopping criteria for tests. Decide before you start: what metric determines the winner, and what minimum effect size matters to your business? A 0.2% lift in open rate might not be worth shipping. A 2% lift in conversion is.

Mistake 5: Running One Test and Moving On

Only 1 in 8 A/B tests drives a significant change. The teams that win build a testing programme, not a single experiment. Consistent, systematic testing compounds — every winning variant raises your baseline and gives you a new, higher starting point for the next test.

---

How Do You Set Up A/B Tests in Customer.io?

Customer.io's A/B testing tools are built directly into campaign and journey workflows. Here's how to use them.

Setting Up an A/B Test in a Campaign

1. Open your campaign and select the message you want to test
2. Click "Turn into A/B Test" — this creates a variation panel alongside your original
3. Set your traffic split — by default, 100% goes to the original. Set your desired split (50/50 is standard for head-to-head tests; 80/20 works well when you're less confident in the variant)
4. Edit your variation — change one element only. Subject line, sender name, body copy, or CTA — not all of them
5. Launch your campaign — traffic routing begins immediately when the campaign goes live
6. Monitor the A/B Test tab — Customer.io displays real-time performance and statistical significance for each variant
7. Declare a winner — when significance is reached (or after your pre-set duration), click "Select winner" to route all future traffic to the winning version

⚠️ Important: Once you end a Customer.io A/B test and select a winner, you can no longer view the non-winning variation's content or historical test data. Screenshot your results and document them before closing.

A/B Testing Inside Journeys

For behaviour-triggered journeys, Customer.io supports a Random Cohort Branch — this is the right tool when you want to test whether sending an email at all vs. sending no email affects downstream behaviour. This is a holdout test, and it's a powerful way to measure the actual causal impact of your messaging.

You can also A/B test individual messages within a journey using the same process above — test the message inside the journey step, not the journey structure itself.

What Customer.io Tracks for You

• Open rate per variant (with MPP caveat noted)
• Click rate per variant
• CTOR (click-to-open rate)
• Conversion events (if you've set a conversion goal)
• Statistical significance percentage
• People count per variant

This connects directly to your segmentation strategy — you can run A/B tests scoped to specific segments, so you're testing what resonates with your active users vs. running generic tests that average across your whole list.

For a complete walkthrough of how testing fits into your Customer.io automation architecture, see our guide on building Customer.io journeys that convert.

---

What's the Right Testing Priority Order?

Not all tests are equal. Here's how to sequence your testing programme for maximum impact:

Tier 1 — High impact, fast results (start here):

• Subject lines → affects everyone, results in 24 hours
• From name → affects everyone, zero design work
• Send time → affects everyone, easy to implement

Tier 2 — High impact, more setup:

• CTA text and placement → directly affects clicks and conversions
• Email length → affects engagement and unsubscribes
• Personalisation depth → affects both opens and clicks

Tier 3 — Strategic, longer timeline:

• Plain text vs. HTML → tests channel fit, not just copy
• Onboarding sequence structure → affects activation and retention
• Re-engagement messaging → affects churn and list health

Tier 3 tests feed directly into your lifecycle email marketing strategy — they're not just optimisation, they're architecture decisions.

---

Email A/B Testing Calendar Template

A testing programme without a calendar is just a collection of good intentions. Here's a quarterly template you can adapt:

Quarter 1: Foundations

Week	Test	Variable	Metric	Sample Needed
1–2	Subject line: short vs. long	Subject line length	Open rate / CTOR	2,000+ per variant
3–4	Subject line: with name vs. without	Personalisation	Open rate / CTOR	2,000+ per variant
5–6	Send time: morning vs. afternoon	Send time	Open rate / CTOR	2,000+ per variant
7–8	From name: company vs. person	Sender identity	Open rate	2,000+ per variant
9–10	CTA: button text variation	CTA copy	Click rate	3,000+ per variant
11–12	Review, document wins, update baselines	—	—	—

Quarter 2: Engagement

Week	Test	Variable	Metric	Sample Needed
1–2	Subject line: question vs. statement	Subject format	Open rate / CTOR	2,000+ per variant
3–4	Email length: short vs. long	Copy length	Click rate / CTOR	3,000+ per variant
5–6	CTA: single vs. multiple	CTA count	Click rate	3,000+ per variant
7–8	Image: with vs. without	Visual content	Click rate / CTOR	3,000+ per variant
9–10	Tone: formal vs. casual	Voice	Click rate / replies	2,000+ per variant
11–12	Review, document wins, update baselines	—	—	—

Quarter 3: Conversion

Week	Test	Variable	Metric	Sample Needed
1–2	Plain text vs. HTML	Email format	Click rate / conversion	5,000+ per variant
3–4	Urgency: deadline vs. evergreen	Framing	Conversion rate	5,000+ per variant
5–6	Social proof: with vs. without	Trust signals	Conversion rate	5,000+ per variant
7–8	Personalised product rec vs. generic	Personalisation	Click rate / conversion	5,000+ per variant
9–10	Re-engagement: offer vs. no offer	Incentive	Re-engagement rate	2,000+ per variant
11–12	Review, document wins, update baselines	—	—	—

Quarter 4: Lifecycle Optimisation

Week	Test	Variable	Metric	Sample Needed
1–2	Onboarding email 1: problem-first vs. benefit-first	Opening hook	Activation rate	3,000+ per variant
3–4	Onboarding sequence: 3-email vs. 5-email	Sequence length	Activation rate	3,000+ per variant
5–6	At-risk: personal check-in vs. feature highlight	Message type	Re-engagement rate	2,000+ per variant
7–8	At-risk: incentive timing (day 7 vs. day 14)	Timing	Re-engagement rate	2,000+ per variant
9–10	Annual review: retest your top 5 winners	Validation	All metrics	Varies
11–12	Plan next year's testing calendar	—	—	—

---

How Do You Document A/B Test Results?

Your testing programme is only as good as your institutional memory. Document every test, even the ones that don't produce a winner.

Minimum test record for each experiment:

Test ID: [Q1-001]
Date: [Month Year]
Hypothesis: Adding the recipient's first name to the subject line will increase open rate
Variable tested: Subject line personalisation
Variant A: "Your onboarding checklist is ready"
Variant B: "{{first_name}}, your onboarding checklist is ready"
Audience: New signups, days 1–3 (n = 4,200 per variant)
Duration: 48 hours
Primary metric: CTOR
Result: Variant B — 3.2% CTOR vs. 2.7% CTOR for Variant A (+18.5%)
Statistical significance: 97%
Action taken: Variant B deployed across all onboarding emails
Learning: First-name personalisation in subject line performs better for this audience segment

Store these records in a shared document, Notion database, or wherever your team tracks experiments. After 20–30 tests, you'll have an audience intelligence library that's genuinely more valuable than any industry benchmark.

---

How Does A/B Testing Connect to Email Personalisation?

A/B testing and personalisation are complementary, not competing. Testing tells you what works for your audience. Personalisation deploys that learning at scale.

The Notion team's success with Customer.io — 49–51% open rates on personalised onboarding sequences — came from combining both. They tested what messaging worked. Then they used Customer.io's Liquid templating to personalise at scale using those tested approaches.

Testing without personalisation leaves revenue on the table. Personalisation without testing is guessing.

---

How Does A/B Testing Affect Email Deliverability?

A/B testing itself doesn't hurt deliverability — but poor testing practices can. Sending large volumes of unengaged recipients inflated by bad test design can damage your sender reputation.

Key deliverability principles that interact with testing:

• Always test on engaged segments first. Don't burn your re-engagement or dormant audiences on experimental messages.
• If a variant produces significantly higher unsubscribe or spam complaint rates, end the test immediately.
• Deliverability issues are often first visible in engagement metrics — which is exactly what good A/B tracking surfaces.

For the full deliverability picture, read our complete email deliverability guide.

---

Frequently Asked Questions About Email A/B Testing

What is the minimum sample size for an email A/B test?

As an absolute floor, aim for 1,000 recipients per variation (2,000 total). Below this, results are too noisy to act on with confidence. For tests measuring click rate or conversion rate (lower baseline rates), you need significantly more — often 5,000–25,000 per variant depending on your baseline and minimum detectable effect. Use Evan Miller's sample size calculator to calculate exactly what your test requires before you start.

How long should you run an email A/B test?

For broadcast campaigns: wait 24–72 hours after send. About 85% of email responses arrive within the first 24 hours, but B2B audiences and time-zone differences mean you should wait for the full 48–72 hour window before calling a winner. For ongoing journey tests (where recipients enter the test over time), run tests for full-week increments to capture day-of-week variation in behaviour.

Can you A/B test inside Customer.io journeys, not just campaigns?

Yes. Customer.io supports A/B testing at the message level inside journeys — you can test a single email within a journey step using the same "Turn into A/B Test" process. For holdout tests (comparing receiving a message vs. not receiving one), use Customer.io's Random Cohort Branch feature. This lets you measure the true causal lift of your messaging against a control group that receives nothing.

Does A/B testing work for small email lists?

Small lists (under 2,000 total) make statistically rigorous testing difficult. You have two options: focus exclusively on the highest-impact, most detectable changes (large subject line rewrites rather than minor tweaks); or accumulate test data across multiple sends over time (treating several identical tests as one larger data set). Avoid making confident decisions based on small-sample tests — the apparent "winner" may just be noise.

What's the difference between A/B testing and multivariate testing?

A/B testing compares two versions of a single element. Multivariate testing compares multiple variations of multiple elements simultaneously (e.g., three subject lines × two CTA texts = six combinations). Multivariate tests require much larger audiences — often 10x the sample size of a simple A/B test — and more sophisticated analysis. For most email teams, sequential A/B testing (one variable at a time) produces more actionable insights with smaller lists.

How do you know if an A/B test result is trustworthy?

Three checks: (1) Statistical significance — was 95% confidence reached before you called the winner? (2) Sample size — did you have enough recipients per variant for your baseline metric? (3) Duration — did the test run long enough to capture your full audience's behaviour? If you can answer yes to all three, the result is trustworthy. Customer.io shows statistical significance in real time, so you can monitor all three without manual calculation.

Should you test on your entire list or a segment?

Almost always test on a segment, not your entire list. Test on a relevant, engaged subset — then roll out the winner to your full list. This protects the broader list from experimental messages that underperform, gives you faster results (engaged users respond quickly), and lets you build segment-specific learning (what works for power users may not work for at-risk customers). Read our segmentation guide for how to structure the right test audiences.

Does Apple's Mail Privacy Protection (MPP) make open rate testing useless?

MPP makes open rate less reliable, not useless. It inflates raw open rates by pre-loading pixels for Apple Mail users (approximately 46% of all email clients). For subject line tests, open rate is still a useful directional signal — the inflation affects both variants equally, so relative differences are still meaningful. But always validate with CTOR (click-to-open rate) or click rate as a secondary metric. For conversion-focused tests, use a conversion event as your primary metric entirely.

How often should you run email A/B tests?

Teams with strong email programmes run 2–4 tests per month. 71% of companies that A/B test emails run two or more tests monthly. There's no upper limit — the Notion team ran hundreds of experiments. The practical constraint is your list size (you need enough traffic to reach significance quickly), your team's capacity to document and act on results, and your ability to isolate variables clearly across concurrent tests.

What's the biggest mistake in email A/B testing?

Ending tests too early. The "peeking problem" — checking results before significance is reached and acting on early noise — accounts for a large proportion of "winning" tests that fail to replicate. The second biggest mistake is testing multiple variables at once, making it impossible to know what caused any observed lift. Set a stopping criterion before you launch, isolate one variable, and commit to the full test duration.

Should you always use a 50/50 split in A/B tests?

Not always. A 50/50 split maximises statistical power and gets you to significance fastest. But if you're testing a radical variant you're less confident in — a completely different tone, a bold new design — start with an 80/20 split (80% to the control, 20% to the variant). This limits your exposure if the variant underperforms significantly, while still generating enough data to evaluate it over a longer period.

What should you test first if you've never run an A/B test before?

Start with subject lines. They're the highest-leverage variable (they affect every open), require the least design work, produce results within 24–48 hours, and give you a clear metric (open rate and CTOR). Test a meaningful difference — not "Your invoice" vs. "Your invoice is ready," but something with a clear hypothesis: personalised vs. non-personalised, question vs. statement, short vs. long. One clean test will teach you more about your audience than any benchmark report.

How do you prevent A/B test results from conflicting with each other?

Run one test at a time on any given audience segment, and keep a test log so you know what's actively running. If you're testing subject lines on your "Active Users" segment this week, don't also run a send-time test on the same segment simultaneously — you won't be able to attribute results cleanly. In Customer.io, structure concurrent tests on different segments to avoid contamination.

Is A/B testing worth it for transactional emails?

Absolutely. Transactional emails — receipts, confirmations, onboarding triggers — typically have the highest open rates in your entire programme (often 50–80%) because they're expected. That makes them ideal for testing, because you have large engaged samples responding quickly. Testing the CTA in a post-purchase confirmation, for example, can meaningfully lift repeat purchase rates with relatively small sample sizes. Notion's team achieved 49–51% open rates on their triggered onboarding sequences and used testing to optimise every step of those journeys.

---

The Bottom Line

James Lind had the answer in 1747. Two sailors recovered. The rest didn't. The data was right there.

The British Navy waited 48 years to act on it.

You don't have 48 years. Your next campaign goes out this week, and the subject line you send will either connect with your audience or get ignored. The only way to know which is to test.

Start simple. One subject line test. One variable. Wait 48 hours. Document the result. Apply the winner.

Then do it again next week. And the week after. Run two tests a month for a year, and you'll have built something the benchmarks can't give you: actual knowledge about your actual audience.

That's what the Notion team built — hundreds of experiments, millions in revenue, and open rates most marketers only see in case studies.

Need help building your testing programme in Customer.io? NerveCentral is a Customer.io Certified Partner. We set up testing frameworks, segment structures, and automation systems that make every send smarter than the last. Get in touch and let's build yours.

---

Citations

1. Lind, J. (1753). A Treatise of the Scurvy. Documented by the National Institutes of Health / PubMed Central. pmc.ncbi.nlm.nih.gov/articles/PMC3536506

2. Customer.io. (2024, July 23). Notion Case Study: How Notion uses Customer.io to drive millions in revenue. customer.io/learn/case-studies/notion

3. Mailmend. (2024). A/B Testing Email Statistics. (Aggregates from 99firms, VWO, Growbo, Litmus, EmailMonday.) mailmend.io/blogs/ab-testing-email-statistics

4. Customer.io Documentation. (Updated: September 17, 2025). A/B Test Campaigns. docs.customer.io/journeys/a-b-test-campaigns

5. HubSpot. (2025). Email Marketing Benchmarks and Open Rate Statistics. blog.hubspot.com/sales/average-email-open-rate-benchmark

6. Mailchimp. (2023, December). Email Marketing Benchmarks and Statistics by Industry. mailchimp.com/resources/email-marketing-benchmarks

7. EmailVendorSelection. (2024). Email Marketing Statistics 2024 (citing Mailjet Inbox Insights 2023, Litmus). emailvendorselection.com/email-marketing-statistics

8. Litmus. (2024). Apple Mail Privacy Protection: How It Changes Email Marketing. litmus.com/blog/apple-mail-privacy-protection

9. Evan Miller. (2024). A/B Testing Sample Size Calculator. evanmiller.org/ab-testing/sample-size.html

10. Bloomreach. (2024). Email A/B Testing: A Complete Guide. bloomreach.com/en/blog/email-a-b-testing