charity: water

Direct Mail Testing

charity: water Goes Mailing?

We did some very cool direct mail testing with charity: water. They’ve agreed to let us publish the results.  Some things worked, others didn’t. We’ll share it all. This will be a two-part post but worth the ride and we’ll also combine both posts into a downloadable, single pdf for ease of future reference.

charity: water is a brand that screams innovation. They aren’t alone in that category but they fit squarely in it. For starters, they signup a ton of monthly donors from a long-form video posted on social and digital ad networks.

And they don’t use direct mail. Ah, the horror! No, it’s a rational choice. Equally rational is their interest in testing the value of direct mail, but not simply as a stand-alone, fulfillment route. They wanted a brand-building, tactile touchpoint adding to engagement and overall value.

And they didn’t want to treat everyone the same. They wanted personalized matching with behavioral science.

Let’s parse that mouthful out a bit.

Brand-building: If you think your direct mail isn’t building your brand and that isn’t its primary job then you should not be in the direct mail business. How so?  The vast majority of people getting your direct mail don’t respond to it.  Does that mean it had no value for the 95% +? It had an impact. It either made those people more or less likely to give in the future and since that’s the massive majority, the impact on that group matters more than the tiny responder group. It builds or destroys donor level equity, which is their future value. That is the sane definition of brand, building future value.

Not in vacuum:  What if a mailing has hardly anyone sending back a paper check?  Did it bomb? In most cases, the analysis would say “yes”, bombed.  But what if you had an extended view that looked at the possible influence of that mail touchpoint on other fulfillment channels, namely digital?

Not everyone is the same. It’s the norm for 99% of sector testing and fundraising to assume everyone is the same.  charity: water is not the norm and needed an approach to personalized matching with rigor and evidence behind it.

Here is the experimental design.

Strategic, Personalized Matching

Tactical Format Tests

Strategic Personalized Matching

It’s a 2×3 design. The Personalized Matching is based on Personality. How did we know who was Open and who was Agreeable?

We have a proprietary, validated set of 3rd party data for each of the Big Five Personality traits. We match each donor to the profiles and assign them based on best fit. Why Personality? It’s mostly innate, fixed and predictive of what we’ll find interesting and relevant. If your copy matches the person’s Personality, they’re more likely to see it, process it, agree with it and act on it.

The QR code test is self-explanatory, there was a QR code on the outer envelope.

The other version had the code on the reply form. Here is that letter and reply.

The post card test was premised on the idea that these are existing CW supporters who, at a bare minimum, are willing to transact online (and may prefer it) and have no need for additional messaging or narrative beyond what a landing page can provide.

Charity Water

The Personalized Matching by Personality had differing letter copy. Everything else was the same – main character, images in the letter, signer, etc.

letter copy charity water

What’d We Find?

There was a holdout group who got the business-as-usual digital marketing but not the mail. And the test groups got what the holdout group got + mail.

First off, we got hardly any paper checks. Shockingly few. In a vacuum, the test bombed, big time. Fortunately, charity: water was as focused on the indirect lift as the direct. So, all results are over a 45-day time period looking at online + offline giving for the respective groups.

Format findings

  • QR Code on Reply & OE is directionally better than on the reply alone. I say directional because rigid significance testing puts the result at 80% confidence and the p-value police will insist on the arbitrary 95% or 90% to flag as ‘significant’.   This result is in line with what we’d expect, make it as easy and convenient as possible and doesn’t require someone to open the OE.
  • Letter beats postcard. Did the postcard get lost in the shuffle of daily mailbox clutter?  Did readers need or want more than what the postcard provided?
    • Sub-story: We did see the postcard do better with Agreeable folks than the Openness group.  This is in keeping with what we’d expect. People high in Openness are not only capable of processing more (interesting) info but they like it.  The postcard probably doesn’t ever match their preference and need.  Agreeable folks want and require and respond to less.

Total Conversion Rate

Agreeable Personalized Matching

Openness Personalized Matching

Personalized Matching Conversion Rate

Personalized Matching Findings

Winning execution Openness, losing on Agreeable. 

The holdout group has no hard, incremental costs to service it – it’s all digital. The net or profit margin for the holdout in the winning Openness treatment is higher than the letter group margin. But, you got a lot more people “engaged” with the brand again; giving, exposure through different channel, etc. by including the mail in the mix.

So, what about digital, couldn’t this same thing be done there to ‘beat’ the one-size holdout control?  Glad you asked and as usual, charity: water was thinking ahead too.

They ran this email test with the last email in a 2-week, country specific campaign.

Higher Donor Conversion

The changes are subtle, an extra line of copy in the Openness (middle) and Agreeable (right) treatments versus Control (left). Plus, copy change tailored to match Personality on the button call out text.

That’s it.  Everything else was identical.

Results?

Mirrors the mail test with Openness doing better than Agreeable but in this digital test they both beat the control albeit only directionally with the latter.

Digital Test

Summary

Test, test, test.  That’s the sector mantra. There ought to be a predicate, think different, then test, test, test. A big part of charity: water success comes from thinking differently, it’s built into their DNA. Not only are they receptive to new ideas, they seek them out.

This testing, like all testing, had constraints. It also had success and failure. All of it contributes to the knowledge base and idea factory for charity: water. That is the real value. Inherent in this is getting beyond just format and channel as the lens onto the world. People are different, we can either market and fundraise to these differences or ignore them. Both are choices and charity: water made theirs.

What will you take from this?