Purpose:
The purpose of this article is to provide information to Outreach Users regarding A/B Testing Sequence Steps.
A/B testing is the easiest way to test changes in your messaging against each other and determine which emails/Templates produce the most positive results.
It is a method to validate that any new design or change to an email in your sequence is improving your open rate, click rate, or reply rate before you make that Template the permanent option.
Intended Audience:
- Outreach Users
Notes:
- A sequence with an Auto or Manual Email Step must be configured to complete this process. For more information, refer to the How To Create a Sequence article.
- Outreach recommends testing small changes, one at a time, with Sequences that target larger numbers of Prospects (such as 100-200 Prospects per Template for a total of 200-400 Prospect emails).
Tracking Open Rates vs. Reply Rates:
A/B Testing is most beneficial when testing a small change against two emails, as adding multiple changes can skew the results and can invalidate the testing. For true A/B testing, you want to test either the Open Rate or the Reply Rate:
- Open Rate is tracked by keeping the same email body and only changing the subject of the email.
- Reply Rate is tracked by having identical subject lines and modifying the body of the email.
If modifications are made to both the Open and Reply Rates, the A/B testing results will not give a clear indication of which is the winner.
How To Create an A/B Test:
- Access the Outreach Platform.
- Click the Sequence icon (paper plane) in the navigation sidebar.
- Sort, filter and click the applicable Sequence.
- Click the Automatic or Manual Email Step.
- Click Add Template and make changes as applicable.
Note: Alternatively, click the More Options dropdown and select the applicable option.
A/B Testing is activated after adding the second Template.
How To Add Prospects to a Sequence with A/B Testing:
When you add Prospects to a Sequence with A/B Testing, you can either have Outreach automatically select which Template the Prospect receives, or you can manually select the Template that you would like to use.
- Access the Outreach Platform.
- Click the Prospects icon (double silhouette) in the navigation sidebar.
- Sort, filter and select the applicable Prospects.
- Click the Sequence button.
- Search and select the applicable Sequence.
- Click Edit Steps in the bottom left corner of the Add to Sequence dialogue.
- Click the Subject Line dropdown and select the applicable change.
- Click Done.
- Click Add to Sequence.
Note: Selecting a specific Template is only available for the first step in a Sequence. Any sub Sequence steps in a Sequence with A/B Testing automatically assigns Prospects to all available Templates in roughly even volumes.
Outreach randomizes which Template a Prospect receives when a Template is not specified. Although it may appear that one Template is being selected by the platform more often than another, Outreach continues to add Prospects to both Templates until even numbers are met. For this reason, it is a best practice to add at least 150 Prospects to a Sequence that you are A/B testing.
Multiple Steps with A/B Testing:
Any Sequence Step that is an auto or manual email can utilize the A/B Testing functionality. However, unlike the first step of the Sequence, you will not be able to designate which Template the Prospects receive in the subsequent step. That is, if Prospect A received Template A in step one, they will not necessarily get Template A on steps two, three, and four. Steps two, three, and four will randomize which Template Prospect A receives.
Tracking Performance
As Prospects go through the Sequence, you'll begin to notice one Template performs better than the other.
You can track performance by looking at the Template statistics on the right side of the sequence step. The report includes opens, clicks, and replies. Hovering over the Template shows the number of Prospects that opened, clicked, or replied. Otherwise the analytics show the percent of emails opened, clicked, or replied based on the total number of email deliveries for that specific Template.
You can disable the Template that isn't performing as well as the other by clicking on the green toggle next to the underperforming Template. A disabled Template is gray, rather than green. This ensures that your Prospects only receive the most effective email Template.
Additional Resources:
The Sales Leader's Guide: How to Write Email Subject Lines that Will Make Your Buyers Reply
Strategy Spotlight: Sequences - Creating the Engine for Your Playbook (1 of 3)
© 2019 Outreach.io