A/B Testing Sequence Steps in Outreach – Outreach Support

Objective

The purpose of this article is to provide information to Outreach Users regarding A/B Testing Sequence Steps.

A/B testing is the easiest way to test changes in your messaging against each other and determine which emails/Templates produce the most positive results.

It is a method to validate that any new design or change to an email in your sequence is improving your open rate, click rate, or reply rate before you make that Template the permanent option.

Applies To

Outreach Users

Notes:

A sequence with an Auto or Manual Email Step must be configured to complete this process. For more information, refer to the How To Create a Sequence article.
Outreach recommends testing small changes, one at a time, with Sequences that target larger numbers of Prospects (such as 100-200 Prospects per Template for a total of 200-400 Prospect emails).
We recommend disabling the 'Allow users to customize step templates when adding to sequence' setting on the sequence ruleset.

Procedure

Tracking Open Rates vs. Reply Rates:

A/B Testing is most beneficial when testing a small change against two emails, as adding multiple changes can skew the results and can invalidate the testing. For true A/B testing, you want to test either the Open Rate or the Reply Rate:

Open Rate is tracked by keeping the same email body and only changing the subject of the email.
Reply Rate is tracked by having identical subject lines and modifying the body of the email.

If modifications are made to both the Open and Reply Rates, the A/B testing results will not give a clear indication of which is the winner.

Create an A/B Test

Access the Outreach Platform.
Click the Sequence icon (paper plane) in the navigation sidebar.
Click the Automatic or Manual Email Step.
Click Add Template and make changes as applicable.
Note: Alternatively, click the More Options dropdown and select the applicable option. A/B Testing is activated after adding the second Template.

Add Prospects to a Sequence with A/B Testing

When you add Prospects to a Sequence with A/B Testing, you can either have Outreach automatically select which Template the Prospect receives, or you can manually select the Template that you would like to use.

Access the Outreach Platform.
Click the Prospects icon (double silhouette) in the navigation sidebar.
Click the Sequence button.
Search and select the applicable Sequence.
Click Edit Steps in the bottom left corner of the Add to Sequence dialogue.
Click the Subject Line dropdown and select the applicable change.
Click Done.
Click Add to Sequence.

Outreach randomizes which Template a Prospect receives when a Template is not specified. Although it may appear that one Template is being selected by the platform more often than another, Outreach continues to add Prospects to both Templates until even numbers are met. For this reason, it is a best practice to add at least 150 Prospects to a Sequence that you are A/B testing.

Multiple Steps with A/B Testing

Any Sequence Step that is an auto or manual email can utilize the A/B Testing functionality. However, unlike the first step of the Sequence, you will not be able to designate which Template the Prospects receive in the subsequent step. That is, if Prospect A received Template A in step one, they will not necessarily get Template A on steps two, three, and four. Steps two, three, and four will randomize which Template Prospect A receives.

Tracking Performance

As Prospects go through the Sequence, you'll begin to notice one Template performs better than the other.

You can track performance by looking at the Template statistics on the right side of the sequence step. The report includes opens, clicks, and replies.

You can disable the Template that isn't performing as well as the other by clicking on the green toggle next to the underperforming Template. A disabled Template is gray, rather than green. This ensures that your Prospects only receive the most effective email Template.