Key takeaways
- Change one variable at a time, or you cannot tell what caused the difference.
- Decide the measure and the sample size before you start, not after.
- Small samples produce winners by luck; treat early results with suspicion.
- A test is only useful if the winning version changes what you do next.
How an A/B test works
An A/B test works by splitting a similar audience at random into two groups, showing each group one version, and comparing a single measure. Running both versions at the same time keeps outside factors, such as the day of the week, from favouring one.
- Write a hypothesis: what you will change and why it should help.
- Choose one variable and one measure, such as replies or clicks.
- Decide in advance how many recipients or visitors each version needs.
- Split the audience at random and run both versions at once.
- Read the result, keep the winner and plan the next test.
A/B testing in marketing and sales
A/B testing in marketing covers anything shown to many people in the same form: emails, landing pages, advertisements, calls to action and sign-up forms. Websites with steady traffic are the easiest place to run it, because samples build up quickly.
In sales, email A/B testing is used on subject lines, opening lines, the offer and the call to action. Outbound volumes are smaller than marketing volumes, so sales teams get more from testing big differences, such as two different angles, than small wording tweaks.
Why small samples mislead
Small samples mislead because with few sends, a handful of results either way can make one version look better by luck. If version A gets three replies from forty emails and version B gets five, that gap could easily reverse next week.
The smaller the difference you hope to detect, the bigger the sample you need. When your audience is small, test bolder changes, run the test for longer, or treat the result as a hint to try again rather than a proven answer.
A/B testing vs multivariate testing
A/B testing compares two versions that differ in one element, while multivariate testing changes several elements at once to find the best combination. Multivariate tests answer more questions in one go but need far larger audiences.
Some teams also run A/B/n tests, with three or more versions of the same element. Each extra version splits the audience further, so for most outbound teams two versions remain the practical choice.
Common A/B testing mistakes
Stopping a test the moment one version pulls ahead is the A/B testing mistake teams make most. Early leads often fade as more results come in.
- Changing several things at once and crediting the wrong one.
- Sending version A to one segment and version B to another.
- Measuring opens when the goal is replies or meetings.
- Testing so many things that no test reaches a useful sample.
Where Pebbles Ai fits
Common questions
How many emails do you need for an A/B test?
Enough that the difference you see is unlikely to be luck, which depends on how big a difference you expect. If your list is small, test bigger changes and repeat the test before trusting it.
Is split testing the same as A/B testing?
Yes, the two names are used for the same method. Some people use split testing more loosely for any comparison of versions, including tests with more than two.
What is the control in an A/B test?
The control is the version you already use, left unchanged so the new version has something fair to beat. Naming one version the control makes clear what question the test is asking.
Does Pebbles Ai run A/B tests on emails?
No. An assistant can draft the versions for you, but Pebbles Ai does not split the send or measure results, so you send them and compare outcomes in your own tools.
About this page. Written by the Pebbles Ai team for people who run sales, marketing and growth. Last updated .
