Key takeaways
- Change one thing per test, or you will not know what made the difference.
- Split the list at random, so both groups are alike.
- Choose the measure first: replies or positive replies rather than opens.
- Small lists need bigger differences between versions to show a clear result.
What should you A/B test in a cold email
Test the parts most likely to change the reader's decision: the subject line, the opening, the main angle and the call to action. Start with big differences, such as the angle and the ask, before you test small wording changes.
- Subject line: a plain topic against a short question.
- Opening: a reference to a recent trigger against a reference to their role.
- Angle: saving time against reducing risk.
- Call to action: a direct meeting request against an open question.
- Length: a short version against a slightly longer one with an example.
How big does a test need to be
A test needs enough responses in each group that a couple of extra replies cannot swing the result. Replies to cold email are rare, so small lists seldom produce a clear winner on fine wording.
If your list is small, test bold differences, repeat the same test across several sends and add up the results. Treat a narrow win as a reason to test again, not as a rule.
How do you run a fair test
A fair test keeps everything but the one change the same: the segment, the send window, the sender and the follow-ups.
Split the list at random rather than alphabetically or by region, which can hide real differences between the groups. Send both versions at the same time, because replies vary by day and hour.
What do you do with the results
Keep the winning version as the new default and test something else against it. Each test should start from the best version so far.
Write down every result, including the losing versions. A note that question subject lines lost twice with finance leaders saves the next person from running the same test.
How to A/B test a cold email, step by step
Decide the change and the measure before you write a word of the second version.
- Pick one element to test and write down what you expect to happen.
- Choose the measure that decides the winner, such as positive replies.
- Write two versions that differ only in that element.
- Split one segment at random into two groups and send both versions at the same time.
- Wait until the follow-ups have run and the replies have arrived, then compare.
- Keep the winner, record the result and plan the next test against it.
Where Pebbles Ai fits
- CapabilityOutreach contentAsk an assistant in chat for two versions of an outbound email with a different angle or opening; generated emails are saved as drafts in your connected Gmail or Outlook mailbox for review.
- CapabilityProspecting & customer recordsTrack the meetings each version produces as deals in the CRM (in Beta), with a note on which version started the conversation.
Common questions
Should I judge subject line tests on open rates?
Not on opens alone. Many mail apps load the tracking image automatically, so opens get counted when nobody read the email; judge on replies where you can.
Can I test more than two versions at once?
You can, but each extra version splits the list further and makes a clear result harder to reach. For cold email, two versions are usually enough.
How often should I run A/B tests?
Run one test at a time, continuously, on your main campaigns. A steady habit of small tests teaches you more than one large test a year.
About this page. Written by the Pebbles Ai team for people who run sales, marketing and growth. Last updated .
