A/B Testing Sequence Steps

Purpose: 

The purpose of this article is to provide information to Outreach Users regarding A/B Testing Sequence Steps. 

A/B testing is a the easiest way to test changes in your messaging against each other and determine which emails/templates produce the most positive results.

It is a method to validate that any new design or change to an email in your sequence is improving your open rate, click rate, or reply rate before you make that template the permanent option.

Intended Audience: 

• Outreach Users

Tracking Open Rates vs. Reply Rates

For true A/B testing, you want to test either the open rate or the reply rate. Open rate is tracked by keeping the same email body, and only changing the subject of the email. Reply rate is tracked by having identical subject lines and modifying the body of the email. If modifications are made to both the subject and body of the email, it will not give a clear indication of whether the subject or body of the email prompted an open and reply to the email. For this reason, we suggest tracking either the open rate or the reply rate, but not both.

Creating an A/B Test

1. To A/B test in Outreach, add the first template to your sequence step.
2. Either click the "+ Add Template" link underneath the original template, or click on the dropdown arrow to the right of the template to easily clone the template for modifications:

You can add as many templates as you want for testing. Once you've added additional templates to the sequence step, you can modify either the subject line or body of the email. Changes can be made by clicking into the second template email. 

Adding Prospects

When you add prospects to a sequence with A/B testing, you can either have Outreach automatically select which template the prospect receives, or you can manually select the template that you'd like to use. 

Note: selecting a specific template is only available for step #1 in the sequence. Any subsequence steps in the sequence with A/B testing will always automatically assign prospects to all available templates in roughly even volumes. 

If you do not select a specific template, Outreach will randomize which template the prospects receive. Although it may appear that one template is being selected by the platform more heavily than the other, Outreach will continue to add prospects to both templates until they are even. For this reason, it is best practice to add at least 150 prospects to a sequence that you are A/B testing.

Multiple Steps with A/B Testing

Any sequence step that is an auto or manual email can utilize the A/B testing functionality. However, unlike the first step of the sequence, you will not be able to designate which template the prospects receive in the subsequent step. That is, if prospect A received template A in step one, they will not necessarily get template A on steps two, three, and four. Steps two, three, and four will randomize which template prospect A receives.

Tracking Performance

As prospects go through the sequence, you'll begin to notice one template performs better than the other.

You can track performance by looking at the template statistics on the right side of the sequence step. The report includes opens, clicks, and replies. Hovering over the template shows the number of prospects that opened, clicked, or replied. Otherwise the analytics show the percent of emails opened, clicked, or replied based on the total number of email deliveries for that specific template.

You can disable the template that isn't performing as well as the other by clicking on the green toggle next to the underperforming template. A disabled template is gray, rather than green. This ensures that your prospects only receive the most effective email template.

Additional Resources: 

Guided Template A/B Testing