Something that I like to talk about when the topic of email testing comes up is the idea that open rates — while the email statistic people are often the most familiar with — are pretty much never the thing you want to be optimizing for.
Why? Simple: You don’t send an email to get people to open it. You send an email to get people to take an action (or, at least, you should!). And the fact is, worrying about open rate too much is not just misguided — it can also be counterproductive.
Here’s a great case in point from recent experience
A couple of weeks ago, for our penultimate appeal as part of our anniversary dinosaur campaign, we did subject line tests on our email.
Splitting our audience of recent donors (past 12 months) into three random test groups, we tested the following subject lines to 5,343 people each:
1) Victory Lap
2) Last chance to get your dino
3) Celebrating with all of you
Here were the open rates for each segment:
1) 49.1%
2) 40.9%
3) 45.7%
But, click rates were:
1) 3.5%
2) 3.8%
3) 3.1%
And here’s the number of donations from each segment — the action we really care about:
1) 43
2) 66
3) 53
So not only did the one doing much worse on opens win in donations, subject three, which was worse than subject one on both opens and clicks outperformed one on the metric that mattered.
It is worth flagging that the win on donations didn’t quite reach the 90% confidence level I’d like, but the broader point certainly still stands.
Earlier this week I gave a training to our new development ($$, not engineering) team here at Mozilla on online campaigns (with emphasis on emails and donations).
I got a few requests for the deck, so here it is!
A few notes:
— This was put together by Matt Compton and me back in 2011 for NOI Bootcamp. I did some updating, but he also deserves some blame. That’s also the reason for the NOI theme on the deck itself. [Edit: bunches of it came from a previously developed training by Colin Holtz. Apologies for the omission!]
— Some of the stats may no longer be correct (eg: avg. reader spends 2-7 seconds on an email), but I don’t think anything in there is so out of date as to substantively matter for these purposes.
— I’ve done my best at Mozilla to practice what I preach, but there are certainly times when you can’t follow every rule. Happy to hear any feedback.
There’s obviously a lot more to cover than just what’s here, but hopefully this is helpful to someone!
Reporting back on our just-completed 2012 End-of-Year (EOY) fundraising campaign: The results were very good!
As compared to last year’s campaign, we did better this year in every way, and finished the year very well: We went from $435,791.56 total raised online in 2011, of which $205,748.20 came after November 28th (when this year’s campaign began), to $751,018.91 raised online in 2012, of which $356,242.69 came after November 28th.
Like last year, this success only came because we were able to get a solid ask in front of audiences who’d be interested in what we had to say. And, again, we saw that the venues which allow the most storytelling and explanation — the emails I sent to our list, and, most importantly, the standalone email to the Firefox & You Newsletter list — were the most lucrative channels, and best ways to fundraise.
Here’s a channel-by-channel breakdown, with the actual numbers in a table below:
| Channel By Channel | 2011 | 2012 |
|---|---|---|
| FF & You Standalone Email | $96,343.79 | $140,067.14 |
| FF & You Newsletter Blurb | $5,525.00 | $10,379.00 |
| Snippet | $51,866.00 | $68,101.63 |
| Mozilla Emails | $21,544.16 | $93,482.13 |
| Misc/Unsourced | $29,214.25 | $30,829.89 |
| Social | $1,255.00 | $3,170.00 |
| Banners | - | $10,212.90 |
Across the board, the numbers were better. What was different?
In some ways, it was just a numbers game: we were able to reach more people directly. For instance, the Mozilla.org list has increased dramatically — we were at ~232,000 subscribers on January 1st, 2012, and closed this year around 580,000, and had a year more in which to tell our story and engage people. Additionally, the Firefox & You list has grown quite a bit.
Further helping matters, the average gift also went up, since the main thrust of this year’s campaign was T-shirts, which require a minimum donation of $30. The graph below shows both donations (blue) & average gift (red), in 2011 vs. 2012.
| 2011 | 2012 | |
|---|---|---|
| Total Donations | 8,428 | 10,221 |
| Avg. Gift | $24.41 | $34.86 |
So where does all this leave us? I’m very happy with this campaign. It was the first time we’ve really successfully fundraised from our email list — $93,000 from 5 emails is pretty awesome, considering where we’ve so recently been, and we continued earlier successes in other channels, most notably the FF & You list and the about:home snippet.
However, there is certainly plenty of room for growth. We’ve consistently done best leading with the “Get a Firefox T-Shirt” ask, but there’s only so many times we can return to that well, and straightforward pitches around Webmaker and other initiatives have not done nearly as well.
Additionally, we continue to have fairly low conversion rates, despite efforts at page testing & optimization, which I think is mostly to do with us still needing to do more in the grand scheme of things to turn Mozilla into an organization where people expect to land on a donation page — and then want to give.
But, lots to smile about with this, and lots more to do! Thanks again to everyone who helped pull this off, and, of course, to anyone reading this who gave.
Since mid-2011, our then non-existent email program has grown by leaps and bounds.
We’ve gone from 26,000 subscribers to 580,000 subscribers, with continued growth of 5,000-10,000 subscribers per week.
We’ve raised well over $140,000 directly over our emails in that period, with more to come in 2012 as our End-of-Year campaign finishes.
And email was a key driver of excitement and growth around the Summer Code Party and the 2011 & 2012 Mozilla Festivals, as well as in our efforts against SOPA & PIPA earlier this year.
However, I think we (I) have fallen down in a couple of key areas:
–Long-term storytelling: I think there hasn’t been a great long-run narrative; I’m not sure that if you signed up for our list in, say, June of this year, you’d really have a great sense of what we’re doing. You’d know a lot about webmaker — from Summer Code Party, MozFest, etc. — but there’s an irregularity to what we’ve been sending that I think has not done a good job of keeping up a steady drumbeat/sense of a movement.
–Trying to apply an advocacy organization model to list growth and engagement: My background has been in working with groups — from Oxfam & NARAL to Obama for America — where taking a pretty classic approach to list growth (eg, petitions and sign-on letters that capitalize on specific moments and move directly into fundraising) has been the right approach.
So I want to change the focus a bit, and concentrate on more frequent, quality content that tells the story of what we’re doing. I’m envisioning roughly weekly, fairly short emails about a single thing within the broad Mozilla/Webmaker/Labs framework that provides a way for people to engage with what we’re doing. This will be in addition to semi-frequent fundraising appeals, of course, but should hopefully give people a much better sense of why they should care, and why a donation to support this work is important.
In contrast to a regular, monthly newsletter (something which has also been proposed internally) this will allow us to keep up a steadier pace and really only send something when we think there’s truly stellar content that warrants it. Regular newsletters run the risk of getting bogged down by the format and by the felt need to shoehorn content in.
Some examples of what these shorter sends could be:
–A description of a really cool event someone put on, that makes putting on an event sound cool and fun to others
–a couple of new projects get added to Webmaker
–The Webmaker gallery launches
–Valentine’s Day Lovebomb
–Something changes about Collusion
–Firefox OS takes a noteworthy leap
–Something else Product-Y that we can offer a bit of promo to
–Awesome standalone graphics (like the Mozilla in 2012 graphic that we sent this week)
–A community input pitch at a key moment
–Etc.
I’m excited to see how it does! I’ll also hold a control group out of this increased volume for a couple of months, to make sure it’s actually helping.
So, as long as you don’t wind up in the control group, look forward to hearing a bit more from us if you’re on our list next year.
And if you aren’t yet, please sign up! Just go to Mozilla.org and enter your address in the email signup box.
There are just a few days left in our contest to win a trip to the Mozilla Festival in London – enter by midnight on Saturday, October 20th, if you haven’t already! – and the first winner has already been picked!
Pulkit Sethi, that winner, is from Washington, DC. He’s a software developer and Summer Code Party participant, who first heard about the Mozilla Festival a few months ago.
In his own words:
I am super physced to be selected to go to Mozilla Festival this year. I think what Mozilla Foundation is trying to do through this event is amazing. Bringing together people from various different domains to collaborate and share their thoughts and ideas on how to educate and empower people…amazing.
Personally for me, what I am looking forward to at the festival is collaborating with others on how to spread digital literacy. When I first started using technology, dial up internet was just coming out and people were still trying to figure out how to collaborate and build the web. But now the ability to learn, collaborate, and share is easier than ever before. I think it’s important to get the next generation involved, and show them how easy it can be to not just be consumers of content, but now become the producers.
I am also really looking forward to collaborating on the intersection of journalism and technology, and getting my hands dirty with a few of the design challenges. Living in DC, I’m inundated with political news and talk all day long. I’m excited to see what ideas and tools we can create to make the processing and understanding of that information easier.
See you in London!
Pretty awesome, huh? And here are a few stats on how the contest is going overall. In short: very well.
Total unique entrants: 14,388
Total entrants via donation: 2,781
Total amount raised: $38,432
And don’t forget: there’s still time to enter for you chance to join Pulkit — and hundreds of other Mozillians — if you haven’t yet!
And if you know you want to come but just haven’t registered yet, get thee to the registration page before tickets run out!
P.S. — Apologies to folks who don’t live in France, Germany, the UK, Mexico, Canada (excluding Quebec), or the US. Legal restrictions on doing contests like this are very onerous, and even being able to do it in these six countries took quite a bit of legwork. Many of the countries where we have the biggest communities simply do not allow chance-based contests like this, others allow them only if they’re based solely in that country, and others simply have rules and regulations that were too much to overcome. We’re hopeful that if we do it again in the future we’ll be able to grow the list of eligible countries.
Later this week, we’re rolling out a fun new fundraising campaign that I wanted to surface before it launches.
If you happen to be on the mailing lists for any of the US election campaigns this might seem familiar to you: we’re making it possible for anyone* to donate for a chance to win a trip to the Mozilla Festival in London from November 9-11. It’s also possible to enter without donating.
This is exciting to me for a couple of reasons:
- I think it will be successful.
- It can help shine a light on some of the awesome stuff that happens at #MozFest every year.
- It’s the first non-evergreen campaign we’re running that’s localized. We have versions in French, Spanish, and German, with participation legal for those in US, UK, Canada (excluding Quebec), Mexico, Germany, and France.
Now, with all that said, a request for you, dear reader(s?): Can you help kick the can tires on our pages before we go wide?
Sample links for each of the four pages are here (with links to the signup pages in the text):
English: https://donate.mozilla.org/MozFest-Contest-Snip
French: https://donate.mozilla.org/MozFest-Contest-FR-Snip
Spanish: https://donate.mozilla.org/MozFest-Contest-ES-Snip
German: https://donate.mozilla.org/MozFest-Contest-DE-Snip
And the QA bug where you could enter anything you see is here: https://bugzilla.mozilla.org/show_bug.cgi?id=790979
A note of caution: These pages are currently live, though this blog will be the first time the links are anywhere in the wild other than bugzilla. So if you make a donation, it will charge you, and if you sign up, it will enter you.
With all that said, thanks in advance for any help or thoughts! If you don’t want to enter something you see, please feel free to leave a comment or get in touch with me directly.
*As mentioned in the bullet list, it’s not actually a fully global campaign at this point — the rules & regs around things like this were too much to make that possible. To enter, you must be 21 or older and live in the US, UK, Canada (excluding Quebec), Mexico, France, or Germany.
