Upload
curtis-poe
View
5.740
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Many people think A/B testing is merely showing customers two versions of something and choosing the one that customers respond to better. It's much, much more complex than that.
Citation preview
P.O.P. Copyright 2013, All Around The World January 14, 2014
A/B TesAng
What your mother didn’t tell you hDp://www.allaroundtheworld.fr/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
What is A/B TesAng?
• TesAng customers, not code • Show different versions • Pick the best one
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Who Relies On A/B TesAng?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Transform Your Company
• Test your customers, not your code • Stop guessing what works
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Should your paycheck depend on my hunch?
Conversion Funnel
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Conversion Funnel
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
→
A Basic Test
• Your current behavior is the base • A variant is a new version (just one change) • You can have one or more variants
• But we assume only one for this talk
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Which Generates More Clicks?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Which Generates More Clicks?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
10,017 Visits 9,983 Visits
125 Conversions 132 Conversions
Which Generates More Clicks?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
9,950 Visits 10,550 Visits
125 Conversions 155 Conversions
The Numbers (95% Confidence)
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
hDp://getdelve.com/2011/12/math-‐behind-‐ab-‐tesAng-‐for-‐curious-‐execuAves-‐part-‐1/
Version Visits Conversion Rate Not Random
Red 10,017 125 1.25% —
Blue 9,983 132 1.32% 51%
The Numbers (95% Confidence)
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
hDp://getdelve.com/2011/12/math-‐behind-‐ab-‐tesAng-‐for-‐curious-‐execuAves-‐part-‐1/
Version Visits Conversion Rate Not Random
Red 9,950 125 1.26% —
Blue 10,550 155 1.47% 94%
Which Sells More Product?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Which Creates More Profit?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
What Confidence Level?
Is 99% beDer than 90%?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
The Cycle
Create Experiment
Only one difference per version!
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Roll the Dice
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Track Performance
Sta:s:cs
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Choose the Best
Pa:ence
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Create Experiment
Wash, rinse, repeat
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Choose the Best
• Ignore the first day • Full-‐business cycle • What if you’re wrong?
• False posiAves
hDp://www.einarsen.no/is-‐your-‐ab-‐tesAng-‐effort-‐just-‐chasing-‐staAsAcal-‐ghosts/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
LimitaAons of A/B TesAng
• Doesn’t tell you why • Side effects • Doesn’t tell you loyalty
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Benefits
• Hyundai: 62% increase in test drive requests • Artsy Editor: 47% increase in click-‐throughs • Visual Website OpAmizer: 50% signup increase • AssessmentDay: 62% increase in sales • SmileyCookie: 41% increase in sales • BaseKit: 25% increase in conversions • Mobal: 27% increase in sales • You Need A Budget: 85% increase in downloads
hDp://visualwebsiteopAmizer.com/case-‐studies.php
What Are They TesAng?
• Add-‐to-‐cart buDon placement • Call to acAon buDons • Using popup forms • Removing content • Link color • Adding tesAmonials • NavigaAon menus • Removing signup forms
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Gemng Started
• Vendor or in-‐house? • Go for quick wins • Run an AA test • Be pa:ent!
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Problems
Ego January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Problems
• Forgemng caching • Non-‐random selecAon
• Forgemng selecAon
• TesAng only the view • Performance
• Dead code • Profit versus conversion
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Tips
• Most experiments fail • Don’t roll the dice prematurely
• A/B test bugs • Test “stupid” ideas • Don’t require rollouts!
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
SegmentaAon
• New versus returning customers • Different browsers • Accept-‐Language • Time of day
• IP country • Gender? Age? Something else?
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
See Also
• The MulA-‐Armed Bandit hDp://stevehanov.ca/blog/index.php?id=132
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
Conclusion
• Lots of money lying on the table • Egos are bad • Data is good • Customer behavior trumps expert opinion
• Customer behavior trumps code behavior
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/
QuesAons?
hDp://www.allaroundtheworld.fr/
January 14, 2014 © 2014 hDp://www.allaroundtheworld.fr/