A/B testing is the most common conversion rate optimization (CRO) method, and is also considered the most valuable. But when testing email campaigns, how long should you wait before deciding a winner? It depends if you’re optimizing for opens, clicks, or revenue, says MailChimp.
To arrive at its conclusions, MailChimp analyzed almost half-a-million users’ A/B tests that featured at least 5,000 subscribers per test. For each test, MailChimp took snapshots at different times and then compared the winner at the time of the snapshot with the test’s all-time winner.
It then looked at how long it took before a test could determine the winner with 80% accuracy, depending on the goal of the campaign.
Open Rates
The results demonstrate that for opens, a wait time of just 1 hour predicted the all-time winner 78% of the time, narrowly under the threshold. As such, MailChimp recommends waiting at least 2 hours, as the accuracy jumped to 83% at that point.
Wait times of at least 12 hours had an accuracy level exceeding 90% when it came to the all-time winner for open rates.
Click Rates
You can pull the trigger a bit more quickly when determining the winner on the basis of clicks, per the report.
In fact, after just one hour an A/B test should predict the all-time winner 83% of the time. And it takes just 3 hours for that accuracy level to hit 90%, after which it continues to climb to 99% after 1 day.
Revenue
You’ll have to be a lot more patient when optimizing for revenue, which isn’t too surprising given the potential delay between an open, click and purchase.
In fact, making a choice based on results obtained in the first hour is more likely to result in a wrong decision than the correct one. The analysis shows that after the first hour the winner of the A/B test (based on revenue) is predicted just 45% of the time.
instead, it takes 12 hours before the winner will be predicted with at least 80% accuracy (82%), and it takes a full day before the accuracy level climbs to 90%.
In summary, the analysis indicates that if open and/or clicks are the goal, winners can be decided within the first couple of hours. But if you’re optimizing for revenue, prepare to be more patient.
The full results can be viewed here.
