Skip to content
April 9, 2013 / Ben Simon

Why open rates are not what you test

Something that I like to talk about when the topic of email testing comes up is the idea that open rates — while the email statistic people are often the most familiar with — are pretty much never the thing you want to be optimizing for.

Why? Simple: You don’t send an email to get people to open it. You send an email to get people to take an action (or, at least, you should!). And the fact is, worrying about open rate too much is not just misguided — it can also be counterproductive.

Here’s a great case in point from recent experience

A couple of weeks ago, for our penultimate appeal as part of our anniversary dinosaur campaign, we did subject line tests on our email.

Splitting our audience of recent donors (past 12 months) into three random test groups, we tested the following subject lines to 5,343 people each:

1) Victory Lap
2) Last chance to get your dino
3) Celebrating with all of you

Here were the open rates for each segment:

1) 49.1%
2) 40.9%
3) 45.7%

But, click rates were:

1) 3.5%
2) 3.8%
3) 3.1%

And here’s the number of donations from each segment — the action we really care about:

1) 43
2) 66
3) 53

So not only did the one doing much worse on opens win in donations, subject three, which was worse than subject one on both opens and clicks outperformed one on the metric that mattered.

It is worth flagging that the win on donations didn’t quite reach the 90% confidence level I’d like, but the broader point certainly still stands.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: