A/B testing subject lines
Test two subject lines on a slice of your audience, then send the winner to everyone else.
An A/B test sends two subject lines to a small part of your audience, measures which one performs better, and sends the winner to the rest. The email body is the same for both; only the subject line changes.
Setting up a test
- Open the campaign and go to the Review step.
- Turn on A/B test subject line.
- Enter Subject B. Subject A is the campaign's normal subject line.
- Choose the test split. This is the share of your audience that receives the test, split evenly between A and B. The rest wait for the winner.
- Choose the winner criteria: most opens, most clicks, or pick manually.
- Choose how long to wait before picking, from 2 to 24 hours.
- Click Send A/B Test (or schedule it as usual).
Setting the split to 100% puts everyone in the test. Nobody waits for a winner; the winner is still recorded on the report for reference.
What happens during the test
- The test slice goes out immediately, half with subject A and half with subject B.
- The remaining contacts are queued as Waiting for winner and are not emailed yet.
- The campaign shows as In Progress until the remainder has been sent.
- When the wait time has passed, Sendra compares the two slices on your chosen criteria, records the winner, and sends the winning subject to everyone who was waiting. Ties go to subject A.
If you chose pick manually, nothing is sent to the remainder until you choose a winner on the report.
Reading the results
Open the campaign's report. The A/B test section shows both subject lines side by side with sent counts, open rate, and click rate. The metric used to decide the winner is highlighted.
While the test is undecided you can:
- Pick winner now to decide early using the current numbers.
- Send A to the rest or Send B to the rest to override the criteria.
Once decided, the section shows which subject won and how many contacts received it. In the recipients table each contact carries an A or B label for the subject they received.
Tips
- Give the test at least a few hours. Opens arrive over time, and a very early pick favours whichever slice happened to go out first.
- Click rate is a more reliable signal than open rate, since some mail clients pre-load images and inflate opens. See Understanding open rates.
- Small audiences produce noisy results. With a few hundred contacts, use a larger split or pick manually.