Upload
fiona-mcphee
View
36
Download
0
Embed Size (px)
Citation preview
Unique test database reveals gold nuggets
Fiona McPhee
Why do we test?
Huge database of test results
No guarantees – nothing stays the same
Short-term vs. Long-term
World of myths to bust
Testing for over 12 years
Keep In Mind
Have a test strategy & hypothesis for each
Does it have a chance of being valid?
Can you do anything with the potential results?
Have a control group & test group(s)
Have a test budget
A / B??
What is affecting response
Digital: What has and hasn’t worked
Channels: Direct Mail, Digital, Phone
What is affecting income
Testing Propositions & Language
What is on the horizon?
% $
The biggest things that influence response rates
Asking
Audience
Contact Rates / Open rates
Ability to contact in future
Should we ask?In a direct mail survey test the
inclusion of an ask at the end of the survey significantly
increased response rate, return on investment and net income.
Test Segment Mailed # Resp Resp Rate Avg Gift Gross Income Net Income ROI
Ask 4847 285 5.9% $49.55 $14,123 -$2,308 0.86No Ask 4855 198 4.1% $57.23 $11,331 -$5,127 0.69
Should we make specific asks?
In a direct mail appeal test we asked in the copy and on the response mechanism against just on the response mechanism. The ask in copy and on the response mechanism significantly increased response rate
Test Segment Mailed # Resp Income Response Rate AvgGift
Ask in Copy & RM 5248 354 $18,818.00 6.75% $ 53.16
Ask on RM Only 5248 299 $17,765.20 5.70% $ 59.42
This is a sample text.Insert your desired text here.
Sample text
The low multiplier significantly increased response rate for a direct mail regular giving conversion ask
Test Segment Mailed # RG Resp RG Resp Rate
Avg Annual RG Value Annual Value
High Ask 2,241 59 2.6% $246.10 $14,520Low Ask 2,242 95 4.2% $171.66 $16,308
How high should we ask?
Audience Testing
Mail: Response highest from most recentPhone: Variables such as channel and pay type also have significant impact
Warm
Strong respondersThere is a cost to swapping
Charity Givers
Mail: Quality of data criticalPhone: Your own leads can increase response
Tepid
Cheap = cheapKnow what you are testing
Cold
WarmWarm
TepidTepid
ColdCold
Charity GiversCharity Givers
Response Rates
Direct Mail Phone Online
Face to Face Media Other0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
Mai
labi
lity
Rate
Contact Rates
Contact RatesCharity A: ~40k RGs Year 1 Year 2 Year 3
# Campaigns / Batches
1 2 5
Conversations 6,960 12,828 19,685Upgrades 2,288 4,840 5,827Yr 5 Net Income $1,086,000 $2,719,000 $2,907,000
Charity B: ~16k RGs Year 1 Year 2
# Campaigns / Batches 1 2
Conversations 5,174 10,901Upgrades 2,070 4,197Yr 5 Net Income $1,141,000 $1,977,000
Contact detail capture: Digital Lead Generation
Mandatory =
Less sign upsHigher quality
90% to call
Not Mandatory =
More volume25% phone opt
out
0
500
1,000
1,500
2,000
2,500
3,000
0%
10%
20%
30%
40%
50%
60%
70%
Addr1 Addr1Email
Addr1Email
Landline
Addr1Email
LandlineMobile
Addr1Email
Mobile
Addr1Landline
Addr1Mobile
Email EmailLandline
EmailMobile
2nd Gift Rate (Year 1) RecruitsBase: 2009 - 2014 Recruits
Multi-contact opportunity donors have the highest
second gift rates. No address sees much lower
conversion.
Address & Mobile potentially an age thing
Information provided by
donors can be increased over
time
Information provided by
donors can be increased over
time
The biggest things that influence income
Asking
Longer is better
Tangible
Asking
Test Segment Qty Mailed Resp Resp Rate Ave Gift Net
Income ROI
Ask 9,831 1,858 19.9% $57.02 $93,610 8.59
No Ask in Copy nor
RM4,240 648 15.3% $47.81 $25,641 5.80
Is longer better?
That longer appeal letters will generate a lower response
8 campaigns – 6 charities2 significant results for response
Only resulted in significantly higher income on one occasion
Test Mailed Resp# RR% AV G Net Return
4pp 9,415 2,554 27.1% $59.03 $124,769 5.80
2pp 9,415 2,287 24.3% $57.44 $106,129 5.21
For an individual test thoughLikely to be marginal difference
Any cost of 4pp vs. 2ppWould have to be less than that.
4pp Generates more large gifts & consistently
more income
Gross Income per ‘000 mailedAn average of an extra $1 per
donor mailed
Is longer better?
Is Tied better?
That asking for Tied funds (vs. General funds) will not increase RR% or Av Gift
Significant result on RR%
Average gift higher for Tied funds but not enough to outweigh lower RR% across the campaign
Test Mailed Response RR% AV G
General 17,324 2,826 16.31% $50.06
Tied 17,325 2,661 15.36% $45.62
“If I can not raise $100,000, I will have to cut X service.”
Vs.“If I can not raise $100,000, we will
have to consider cutting the X program and others like it.”
More Gold?World largest single mass of Gold, 630 pounds
Testing Propositions
Three propositions vying for supremacy
Two taken to the phones with warm donors
One significantly out pulled the other half way through calling
01
02
03
Welcome to Stroke Foundation Life Savers.
Thank you for taking the time to speak with one of the team recently and thank you so much for making the wonderful decision to join Life Savers.- helping us to run a powerful and sustained national campaign to stop so many people dying needlessly. Together we can save a life every hour.
Thank you for taking the time to speak with one of the team recently and thank you so much for making the wonderful decision to join us in spreading the message to save lives - helping us to fill in the gaps in people’s knowledge about stroke.
By choosing to make a monthly donation you are continuing to change lives and will be helping to make sure everyone in New Zealand knows how to reduce the chance of stroke, how to recognise stroke and what to do if a stroke happens.
Testing Language
Does the use of colloquial language increase response rate?
The use of Kiwi (vs. New Zealand) language in variable copy increased response rates. Average gifts were down for the colloquial test group – did we put off posher donors?
Less formal language positively impacting response. Finding the right tone in relation to each appeal / story important and we are looking at testing/adaptation for the Mid-Value Donor program.
01
02
03
Test Segment
Mailed Cash Avg Gift RG Avg Gift Comb
ROIComb RR%
Kiwi 26,534 $46 $13 3.04 4.21%
New Zealand 26,454 $57 $14 3.04 3.83%
Digital: Direct to Donation
10 years ago it worked
Early adopters getting results
Integrating warm was getting traction
Asking to get married before the first date?
Digital: Direct to Donation
Too high a barrier to
donating as first interaction
10 years ago it worked Now: Wall to Wall
advertising
Lifeline
STEP 1 – Drive to Petition
Facebook – best performing creative, Step 1 – Petition
1 2 3
Lead AcquisitionSignatures: 65,027
Email opt-ins: 31,609 Phone opt-ins: 8,838
Some of the things we are testing in the digi space now
• Digital to phone• Digital to digitalConversion Step• Petition• Survey• Pledge
Lead Sources
• Targeting within mediaMedia• Link Ad• Canvas• Carousels• Video Slideshows
Format• Stories• Images• Subject lines
Creative
Channel Integration
Does the provision of online options increase survey response?
Including both the launch email and the post email had the most significant impact on Survey response. The emails had no significant impact on financial donations to the Survey.
Email integration had a positive impact on the key objective of the appeal. Continuing to increase email penetration rates and testing email execution needed.
01
02
03
Ask Tests
Level change against each
other
Fixed string against
calculated
Number of asks
Type of asksCombining NL/Survey with an ask
Highlight asks
Reversing sequence
Other than ask1 in copy,
no ask in copy
My choice on RM
Having no calculated asks on RM and copy suppresses the response and results in lower average gift
Numbers of asks in copy – insignificant outcomes
Adding ask at end of survey increases
response rate
5 Charities, 6 tests, 1 significant
Number of Asks
Ask Testing
Sample text
Highlight asks
Insignificant
Reverse Sequence Ask 2 As Main
Lowers Response but appears to increase
average gift
Insignificant Significant
Direct Mail Pack Additions Testing
Does inclusion of a premium (Note pad) vs no premium uplift response rate?
Inclusion of a Notepad Premium uplifted response rate to the appeal – however due to the higher cost of this pack, ROI is lower
Premiums continue to uplift response, however cost and audience needs to be carefully balanced to make it a valid option going forward
01
02
03
Pack Additions Testing
Significant higher response rate and net income
Include a Calendar
Significantly lower average gift
More responded
Significantly higher response rate and average gift
Include an Involvement Device
Significantly lower net income
Sample text
Significantly higher response rate
Post It Notes
Increased net income
Significantly lower response rate & net income
Adding Newsletter
Higher average gift with newsletter
More responded More responded Less responded
What is on the horizon?
1
2 Splitting incorrectly & not sticking with it
What is it? How do you provide it? How can you test its impact/value?
Measure the income & retention vs. total cost over an extended period of time
3
Stewardship / Donor Care
Wrong
Right
What is on the horizon?
1
2 Are we under emailing? Under mailing? Under communicating
What is appropriate? What is effective?
One channel / more channels? Or is it all too much?3
Frequency of Contact
Too Little?
Too Much?