Embed Size (px)
Citation preview
A/B Testing – can you afford not to?#13NTCab
Hugh Dwyer,International Rescue Committee
Jacob Colie,Stage Coach Digital
Tom WilliamsonJackson River
Today’s Agenda
• Introductions
• What is A/B Testing?
• Testing Goals – what should we test?
• Testing versus Experimenting
• Crafting an A/B Test Plan
• Selling your plan
• Build, Measure, Learn
• Testing Tools
• Resources & Downloads
• QA
Slide 2A/B Testing
What is A/B Testing?A/B Testing:
A/B testing is an experimental approach to web design which aims to identify changes to web pages that increase or maximize an outcome of interest (e.g., donation conversion). As the name implies, two versions (A and B) are compared, which are identical except for one variation that might impact a user's behavior.
Slide 3A/B Testing
Standard Donation Page
Slide 4A/B Testing
A/B Test Donation Page
Slide 5A/B Testing
What is A/B Testing?Multivariate Testing
Multivariate testing is similar to A/B testing, but tests more than two different versions at the same time.
Slide 6A/B Testing
Multivariate
Slide 7A/B Testing
ChallengesWhat are some common challenges to doing A/B Testing?
• Resources – technology, time and people
• Data sampling size
• Is my list big enough?
• Are my results significant?
• “Do no harm”
• Lack of existing test data to build on
• Risk vs. Reward
To do A/B testing you need to have an Entrepreneurial Spirit!
Slide 8A/B Testing
A/B Testing ChannelsChannel Testing
• Advertising
• Web
• Offline/Direct Mail
• Integrated (On and Offline)
• Segmentation
Slide 9A/B Testing
Testing Goals – what should we test?
Channels and Audience Segments
Slide 10A/B Testing
Snail MailEmailSearch
Website/Landing Environment
Advertising
New Audiences
Leads
New Donors
Multi-Year Donors
Testing Goals – what should we test?
Metrics and KPIs
• Establish benchmark metrics: what are you trying to beat?
• Average time on site/page
• Open rates
• Click-through rates
• Form conversion (donations, email signups, etc.)
• Average Gift
• Cost to acquire
• Cost per lead
Slide 11A/B Testing
Testing Goals – what should we test?
Evaluating options
Slide 12A/B Testing
• Alignment with strategy, context
• Examine conversion funnels and donor lifecycle: where are your challenges?
• Consider potential returns, & calculate ROI
• Where can you move the needle the most?
• Quick wins and low hanging fruit
Testing Goals – what should we test?
Variables
Slide 13A/B Testing
Design elements
Persuasion assets
Offer elements
Benefits
Default ask
Products
Price points
Ask string
CTA
Multi-step forms
CTA Positioning
ImagesCopy
Media mentions
Premiums
Security seals
Testimonials
Achievements
Page layout
Expert ratings
Subject lines
Video
Keywords
Communications strategies
TimingFrequency
Cross-channel
Usability
Modal windows
Content
Testing Goals – what should we test?
Considerations
• Evolutionary vs. revolutionary changes
• Frequency and pace of testing
• Short-term vs. long-term
• Easy road vs. hard road
• Embrace (calculated) risks
Slide 14A/B Testing
Testing vs. EXPERIMENTS
Slide 15A/B Testing
Slide 16A/B Testing
When to test? When to experiment?
Slide 17A/B Testing
GOALSAccelerate your REVENUE
Provide VALUE for your users
Slide 18A/B Testing
The DIGITAL REVENUE CYCLE
BUILD. MEASURE. LEARN*
Slide 19A/B Testing
* Read THE LEAN STARTUP
BUILD an MVP
Slide 20A/B Testing
MEASURE revenue andengagement
Slide 21A/B Testing
LEARN what did and didn’t work
Slide 22A/B Testing
Baseline
IDEAL
Slide 23A/B Testing
Slide 24A/B Testing
vs.
short-term vs. LONG-TERM
Slide 25A/B Testing
5% vs. 35%
Slide 26A/B Testing
Examples
Slide 27A/B Testing
MERCY CORPS Retention
Slide 28A/B Testing
IRC Rescue Gifts
Slide 29A/B Testing
Getting started with A/B Testing• KISS – Keep it simple silly
• Define metrics to test
• Sustainer Acquisition
• Average Gift
• Gift String
• Conversion %, etc.
• Identify your top 3-5 items to test, and schedule them over X months!
• Decide how you will test each scenario
• Control group
• Minimum data required to validate
• Resources necessary
• Internal communications
• Implement each test
• Build, measure, learn – be patient
Slide 30A/B Testing
Individual Test Cycle
Slide 31A/B Testing
Identify KPI/Metric
How will you test?
Coordinate ResourcesImplement
Measure, Report, Analyze
Selling your plan
Slide 32A/B Testing
Objectives Inputs Plan
Developing a business case
Analysis Recommendation
Resources & Downloads
Presentation Materials
• PowerPoint - www.jacksonriver.com/nten/ab_testing
Third Party Tools:
• Optimizely - www.optimizely.com
• Visual Website Optimizer - http://visualwebsiteoptimizer.com
• Google Content Experiments
• 8 Seconds - http://www.8seconds.net
Slide 33A/B Testing
Questions and Answers
Contact info
• Hugh Dwyer
• Sr. Director, Information & Operations, International Rescue Committee
• Jacob Colie
• Senior Strategist & Principal, Stagecoach Digital
• @jacobcolie
• Tom Williamson
• CEO, Jackson River
• @tomwilliamson, @jacksonriver
Slide 34A/B Testing
Evaluate This Session!Enter for a chance to win an NTEN engraved mini iPad!
Or, search by session title at www.nten.org/ntc/eval
