The key to A/B split testing in email is patience. Wait as long as you can to analyse results and you’re more likely to get a true picture of what will happen when you roll out your eventual email campaign.

Does your ESP provide split testing?

Marketers and ESP sales teams often sell email deployment tools with built-in A/B split testing software on the basis that they will make your programs run with the minimum amount of resource.

The A/B split testing functionality varies by platform, from running the entire split test and deploying the “winning” campaign automatically, to simply splitting the data into 2 random sets.

As with most tools, A/B split testing software tends to be generic. If you have a specific need or want to test something that is unique to your organisation, then the manual approach is often the best method, and it’s not half as difficult as you might think…

How do I go about manual A/B split testing?

To ensure that your findings are robust and easy to implement, start with your test plan. Once you’ve decided what you’re going to test, there are 3 main areas you need to consider: data, set up and analysis.

How do I split my data?

To test the impact of one variable on another, a random selection is the best choice for most marketers – essentially splitting your list in half.

Only have a small amount of data? Plan on breaking the analysis down into smaller groups, such as demographic groups. Make sure that you have equal and sufficient representation in each group.

Alternatively, if you believe that one approach is likely to deliver significantly higher returns, you may choose to only keep a small sample for one variable, and send the majority of addresses the alternate option.

How big does a representative sample need to be?

The answer depends on a number of factors:

• primarily, how statistically viable you need the results to be
• your objectives
• how many factors you are testing

There are differing opinions on sample size, but it is smaller than you would think. The sample size used to predict the general election is based on just 2000 people, for instance, although over 27 million people actually vote.

You can choose your sample size scientifically by using a sample size calculator. You can find one easily – just type “sample size calculators” into a search engine.

How to use a sample size calculator:

Say you wanted to test for how many people would open your email if you sent it to 10,000 people and you wanted the accuracy of your answer to be within 5% either way. That means your final answer is going to be an interval, like 25-35%.

So how many people do you need to sample? Here’s how to find out:

1. Find a sample size calculator online, like this from creative research systems or this from Raosoft
2. Choose the size of the interval you want as your answer – in this example it’s 5%
3. Type in the population – in this case it’s 10,000
4. Choose how certain you want to be. You can never have total certainty, but most calculators let you choose 95% (pretty sure) or 99% (very sure, which means you’ll need to use a larger sample)
5. Press calculate and the calculator will give you your sample size. For this example you can be pretty sure by sampling 624 people.

So if 10% of your sample open your email, you can be pretty sure that you’ll get a 5-15% open rate when you send it out to 10,000 people.

How do I set up my mailings?

Once you’ve split your data, all you need to do is set up your mailings.

If you plan on delving deeper into your results to profile different segments, the best way may be to set the messages up as one campaign and analyse the data offline after the campaign has finished.

Though this approach requires some additional data work initially, it will keep your campaigns organised within your platform and ensure your future regular campaign reporting remains consistent.

You could opt to:

• test different subject lines, content or creative by using dynamic content
• test a variety of send days by setting up a triggered campaign, based on a field you add to your data
• determine the most effective deployment time, if your list is large enough, by implementing your campaign very slowly

Extra tip: track how you split your campaign and make sure that you can identify which subscribers received which test.

How do I analyse the results?

It can’t be stressed enough: you must let your campaign run its course. We’d advise any results gathered in less than 2 days will probably turn out to be inaccurate as the testing continues. Depending on your purchasing cycle, you may need to wait as long as a month. Then, gather as much information to include as you can – this can be anything that you can attribute to an email address, such as revenue, or even web stats.

To determine the impact of each of your tests, you can then layer this information over your sent, delivered, opened and clicked activities for each email address.

Based on your hypothesis, you will then be able to easily see which campaign met your objectives best. Look at all possibilities. By splitting the data in different ways (which is easier to do offline), you may uncover some surprising findings that could guide future testing or campaign development.

Clearly defined testing programmes are the benchmark of accurate, finely-tuned email campaigns. They help improve the revenue you can generate from email marketing – and that’s the ultimate aim after all.

These 10 handy tips will help you develop your testing programme effectively. Implement them into your testing activity and see the difference it makes…

1. Make use of your historical data

Increase your open rates by using historical data. For example, if you’re trying to find out which days of the week have the most impact in terms of open rates, check your old data. Chances are you’ll have already mailed out on 5 out of the 7 days and can judge when your next emails should go out to maximise impact.

2. Work out what you want to test – develop a hypothesis

Define what you are testing for. This could be:

• open rates
• click rates
• purchases
• revenue
• unsubscribers

Then decide the factors you want to know the impact of. These could be:

• day of the week the email is sent
• week of the month the email is sent
• subject line length
• common words in subject line
• types of content
• message frequency

3. Compare like for like…

Attribute differences to the specific factor you are testing by splitting your testing across multiple campaigns. Testing more than one factor on any one campaign will confuse results.

4. Take on common assumptions

There’s plenty of material out there to tell you what factors are influential, including research, published articles and case studies. By all means review these, but try testing influential factors yourself.

No two brands have the same market, product offering and subscriber base, so don’t miss out on genuine opportunities for improvement!

5. Segment your list into groups who behave in similar ways

While it would be easier and more profitable if your subscriber base had the same behaviour patterns for purchase and engagement in your activity, they simply don’t.

You can analyse the size and value of subscriber groups and calculate the point where developing campaigns for new groups fails to generate a worthwhile return by breaking your mailing list down into groups – through interests, spending or interaction patterns.

6. See the campaign through and then analyse the results

We always recommend splitting your campaign into test cells, then analysing the results well after the activity has finished. Usually two weeks after is a good starting point, but a month is better, depending on your brand and subscriber purchasing patterns.

7. Include revenue results for a full, complete picture

Opens and clicks are very useful and provide a good level of guidance, but for most marketers the main objective is to generate sales. Make sure you include revenue in your test period analysis.

8. Don’t expect quick wins

The best way to approach testing is with a long term focus. Make small, regular changes to your campaigns and then look at the improvements after a longer period. This will give you more reliable results. Don’t start trying to double your revenue by tweaking your template!

9. Think outside the box (of your campaign)

Use factors outside the scope of your activity, like the impact of the position of your opt-in / subscription link.

10. One last thing… NEVER stop testing

It’s a fact of life – things change over time. Your competitors, customers, general spending patterns and everything in between will alter and evolve. Even if you tested a certain factor a year ago, test it again to make sure it’s still the right approach.

With these top 10 tips under your belt, you’re all set to test effectively. Good luck!

I need to do an email test – tell me a bit about A/B split testing.

The A/B split email test is the classic way email marketers test the performance of their new email activity. It looks at 2 different variables of email response, typically subject line, content or creative.

Most often the test will look at an email’s performance for just a short time period, to see which subject line / creative treatment etc achieves the best open and click rates. But at Alchemy Worx, we’ve seen that isn’t always the best way to assess which approach will be most successful overall.

So what’s the issue with A/B split testing?

The problem is that the short time period distorts the result. Again and again, our research shows that the test cell in the lead after a few hours rarely ends up as the most best performing cell after a few days, especially when sales are taken into account.

This is because the people who interact with an email first are a very specific subset of your customer base – very different to (and commercially less significant than) the sort of people who will interact with an email over an extended period.

Of course, there will be some people will be actively waiting for your email and will open immediately – but this is not typical behaviour. The usual behaviour is not to open the email at all, meaning there are no guarantees that the people who act first are going to act the same way through the length of the campaign.

Furthermore, although A/B split testing may uncover the best “average” approach, the reality is that there is no such thing as an average subscriber. When analysing your data, you may find that some things work best for some users, whilst another approach works better for others.

So how else can I test the impact of different factors?

It’s not as sexy, perhaps, but the most effective way to assess a new email campaign’s performance is to crunch historical data.

After all, a test – however sophisticated – is only a prediction. Analysing historical data, on the other hand, is to look at the factual record of what actually DID happen – and that should be the basis of analysis for future testing.

So use your historical data to check what you have already tested. You will most likely have sent mailings at different times of the day, different days of the week etc. Use that knowledge to fine-tune your future test plan, and only test what you need to test.

Analyse your individual subscriber’s behaviour to group subscribers into segments of people who respond in certain ways. For example, it may be worthwhile developing an additional program for your active subscribers.

Top Tips
Some tried-and-trusted methods to reinvigorate interest in your email activity:

• Make the most of what you already know about your subscribers by analysing your historical data.
• Consider grouping subscribers into segments according to response.
• If you’re A/B split testing, make sure to leave enough time to get truly representative results.