When working with Shelter to develop three donor welcome journeys, we set up a number of tests and monitoring for future optimisation.
Now the results are in and we’re happy to share with you insights on what worked, what didn’t and how we used this real performance data to refine the journeys even further.
More Onion are just incredible. They’re brilliant at challenging you, steering the conversation and taking a pragmatic approach that always puts the audience first. Their collaborative approach to strategy development helped us to deepen buy-in and alignment from key stakeholders from across the organisation.
Shelter brought us on board to create three welcome journeys for their individual giving supporters - one for one-time gifts, one for monthly donors and one for lottery players. You can learn more about these projects here:
As with all welcome journeys, these were set up so that we can learn from them and make improvements over time to boost their performance even more. All three journeys were set up with journey-wide A/B tests as well as monitoring on individual content to see what was performing well and where there was potential for improvement.
In this case study, we’ll share the results of the tests and other analysis and how we used this to improve the journeys.
If you'd like to learn more about A/B testing, you can download this free report on optimisation.
Journey objective: Player retention
The first version of the More Onion lottery welcome journey boosted six-month retention by ~9%, translating to a projected £250,000 in additional income over five years.
This was a fantastic start, but there is even more potential available to boost retention and create incredible long-term income gains. So we looked to performance and testing data to see how to refine the journey even further.
We were looking to better determine why people play the lottery. For personal gain or because they care about the cause?
The six-month journey was split into three test variants into which supporters were randomly assigned. One focused on personal benefit (you could win), one on cause benefit (here's the impact of supporting Shelter), and one more evenly balanced these two themes.
The cause-focused stream performed best for retention, but only to a 76% level of statistical significance. This finding was reinforced by survey data and audience research which showed that players were really motivated by the impact they were having on the housing emergency.
We also learned that lottery players are more likely than other supporter groups to have personal experience of a housing emergency, and that they respond strongly to positive impact and first-person storytelling.
We’ve built an updated version of the journey that focuses more on cause-led content, including adding richer impact storytelling, particularly first-person stories, and bringing positive stories of impact to life with interactivity and art.
We’ve also made sure that the journeys signpost to Shelter's resources for helping people who are having difficulties with housing.
Finally, we extended the journey from six to twelve months to close a retention drop-off that was happening after the welcome series ended.
Journey objective: The donor makes a second donation.
From the initial journey project, we saw a 50% increase in second-gift rate for email journey recipients compared to a pre-journey benchmark for all cash givers. Important to note here is the huge performance discrepancy between supporters who received the email journey and those who didn’t (because they didn’t opt in for email).
Supporters who gave contact consent (email and/or post) were twice as likely to make a second gift as those who didn’t. This shows not only the big impact that the welcome journey was having but also the importance of optimising contact consent opt-ins at the point of signup.
If you’d like to learn more about collecting and managing contact consent, you can download this free report on optimising your email opt-ins.
We wanted to know the ideal gaps between the emails - can we send the emails with smaller delays between them without negatively affecting the journey’s performance?
We split the audience into two groups with identical content, varying only the pace: one group received emails every ~8 days over six months, the other every ~6 days over five months.
We found that there was no statistically significant difference between the performance of the two versions — meaning we can send faster without hurting second-gift rates, which is useful for moving donors into Shelter's business-as-usual stream sooner.
From email analysis we also learned that attention is highest at the start of a journey, quality first-person storytelling consistently outperforms other formats, and campaign content needs rewriting to appeal specifically to donors.
We’re front-loading engagement actions in early emails to capitalise upon and maintain attention. We’re adding more first-person storytelling and we’re rewriting campaigning content with this specific audience in mind.
Finally, we’ve set up new tests on using SMS as a complementary channel to boost donations.
Working with More Onion was very collaborative, they act like part of the team and we really saw them that way. They’re generous with their knowledge and we learned so much from their expertise. It's my ideal way of working with an agency.
Journey objective: Long-term donor retention
For this journey we wanted to find out - do we retain more supporters if we communicate less (i.e. mostly leave them alone), or by working to actively engage them in Shelter’s work e.g. through non-financial actions?
To find this out we built two versions of the six-month welcome journey: one with loyalty/engagement content only, one with additional campaigning asks. As with all tests, supporters were randomly assigned to one of these experiences and then we compared the difference in performance of the two versions.
This test yielded no statistically significant result - essentially, there was no notable performance difference between the two tested journeys. But the performance analysis of the emails, pages and survey data showed some very interesting things…
People who engaged with emails were much more likely to keep giving. But there was seemingly also a correlation between receiving more emails and early cancellation.
This data indicates that there are two distinct types of donors, with very different needs: those who want to feel involved and connected, and those who simply want to give quietly and be left alone.
Additionally, individual email analysis also showed that this audience responded well to a thank you offering (an art print) and engagement actions.
If you’d like to learn more about engagement actions, you can download this free report on deepening relationships with engagement actions.
Our biggest challenge with the second iteration of this journey was identifying who belongs to which group (active involvement vs fewer comms) and then meeting these two different supporters' needs.
To achieve this we designed the updated journey with lots of engagement opportunities early on, such as mini surveys and interactive content. These early and easy opportunities to engage should allow us to identify whether individual supporters prefer to engage or to receive less.
Donors who engage with early content get nine emails over six months with rich storytelling and impact content. Donors who don't get five. We also made sure that among the content sent to everyone were more opportunities to engage, so that even if people didn’t engage in the initial weeks, they could identify themselves later in the journey as enjoying more content and involvement.
Always make sure you have a plan to review and improve your welcome experiences. Even small percentage increases in retention or conversion quickly add up to big money.
When we couldn't prove one version beat another, that itself was information — and the wider journey data always filled in the picture.
Consider ways to make the stories stand out, such as first-person storytelling, interactive storytelling and using visuals. This is motivating for all types of donor.
We'd be happy to schedule a time to speak.
