Naked Email: Putting Subject Line Pro to the test and what we found looking at opens over time
.svg)
"How did an 18-year old intern beat an Alchemy Worx subject line expert in a split test?"
You might think there isn't much 'Naked' about a run-of-the-mill subject line split test but this one was a little different. Here's why...
We decided to run a simple subject line split test on our first Naked Email. We weren't concerned with the content of the subject line, only in the methodology used to construct them. One subject line would be written by a human, the other by a machine.
The Match-up:
In the blue corner representing "Man" we invited our very own Dela Quist, widely regarded as one of the world's leading authorities on subject lines.
In the red corner representing "Machine" was Subject Line Pro, the subject line prediction tool that analyses over 26 billion emails at the touch of a button.
The Rules:
Each party was given the email and background on the user base
The email strategist was only allowed to use their intuition and experience
Subject Line Pro was used by the most inexperienced member of our staff, an 18-year old summer intern. All the intern had to do was start with the expert's subject line and optimize it using Subject Line Pro until their subject line was predicted to win.
So who won? Man or machine? Expert or novice?
The results: As you can see, Subject Line Pro was the clear winner in this test with a confidence level of 99.7% for opens and 98.9% for clicks.
We will leave you to draw your own conclusions as to the "why" but this highlights the important role data plays in what is often seen as an intuitive discipline. It's just a bit of a pain having to crunch the numbers every time you want to write a new subject line. That's where a tool like Touchstone is such a valuable addition to the subject line writer's armory.
Our Head of Copywriting, Dan Hamilton, commented: "It's removing the subjectivity from subject lines. The data will always give a straightforward factual answer and no matter how good a writer you are, you need to work with the data. I've been using Subject Line Pro for some time and this really confirms my view that it is an indispensable tool for putting subjective judgments aside and gaining a better understanding of what makes a winning subject line."
Start Driving Engagement with Subject Line Pro
Save time and improve results with virtual subject line testing that delivers real, accurate results in seconds. Get as much as a 21% lift in performance!
Digging deeper:
A phenomenon we've regularly observed in subject line split-tests is how the result changes over time. Since this test produced such a robust result, we wanted to see how the results over time measured up. Looking at the increase in the open and clicks over 48 hours, then day-by-day, the relationship appears to be pretty steady, with a gradual widening between the A and B results over time: However, looking more closely at the diversion between the A and B opens over time shows there's more to this first chart than meets the eye:
As we've long maintained, subscribers react differently to subject line tests over time. The overarching take-out is that the diversion between the two open rates increases over time. But it's not a smooth line, which is why we decided to plot the chart by time-of-day rather than hours-after-send to get a more detailed picture of what was going on with this mailing.
We know from the first chart that around 50% of all opens occurred within the first hour, but the difference between the A and B versions was only 5% by this point, growing to a 12% difference by 8 p.m., about 5 hours later. Between 8 p.m. U.K. time and 8 a.m. U.K. time, the test proceeded with the two versions running at a fairly consistent 11% difference but at 8 a.m. there was a surge of opens and the A version jumped to about 18% ahead of the B version. During the day the B version made up some ground, with the eventual difference leveling off at around 15%. The 'V' that happens along this otherwise steady line is on a Sunday - we didn't get very many opens on a Sunday, but there was a slight difference in results on that day.
This raises some intriguing questions about time of day and time-zone. For example:
If we had sent the email earlier in the day and gained proportionately more of our opens and clicks on day 1, would the gap between the A and B open rates been smaller?
Can a subject line be optimized for time-of-day?
Our list has global reach but is mainly UK and US subscribers - do they respond differently to subject lines? Is one group generally more active than the other?
Watch out for upcoming Naked Email posts to find out how we get on pursuing these questions!
Schedule a Call
