View
258
Download
3
Category
Preview:
Citation preview
THE EVOLUTION
OF A/B TESTING
@Nicole_Mintiens
Data Analysis & Insights Manager @GetG5
41 Shades of Blue
The Evolution of A/B Testing
@Nicole_Mintiens
$200 MILLION
The Evolution of A/B Testing
@Nicole_Mintiens
The Evolution of A/B Testing
@Nicole_Mintiens
Advertising and Marketing• Improved ROI = Good
The Evolution of A/B Testing
@Nicole_Mintiens
Advertising and Marketing• Improved ROI = Good• Wasted Resources = Bad
The Evolution of A/B Testing
@Nicole_Mintiens
Advertising and Marketing• Improved ROI = Good• Wasted Resources = Bad• Marketing = Complex
The Evolution of A/B Testing
@Nicole_Mintiens
Advertising and Marketing• Improved ROI = Good• Wasted Resources = Bad• Marketing = Complex• Facts > Opinion
The Evolution of A/B Testing
@Nicole_Mintiens
WHY SPEND TIME TESTING?• $92 traffic /$1 optimization
Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014
The Evolution of A/B Testing
@Nicole_Mintiens
WHY SPEND TIME TESTING?• $92 traffic /$1 optimization• 87% run 1-5 tests/month
Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014
The Evolution of A/B Testing
@Nicole_Mintiens
WHY SPEND TIME TESTING?• $92 traffic /$1 optimization• 87% run 1-5 tests/month
– Best brands run hundreds per month
Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014
The Evolution of A/B Testing
@Nicole_Mintiens
A/B TESTING FACTS• Improves conversion rate
The Evolution of A/B Testing
@Nicole_Mintiens
A/B TESTING FACTS• Improves conversion rate• Enhances effectiveness
The Evolution of A/B Testing
@Nicole_Mintiens
A/B TESTING FACTS• Improves conversion rate• Enhances effectiveness• Optimizes ROI
The Evolution of A/B Testing
@Nicole_Mintiens
A/B TESTING FACTS• Improves conversion rate• Enhances effectiveness• Optimizes ROI
So… YOU should test!
The Evolution of A/B Testing
@Nicole_Mintiens
The Evolution of A/B Testing
@Nicole_Mintiens
MISCONCEPTIONS• Hurt conversion rate
The Evolution of A/B Testing
@Nicole_Mintiens
MISCONCEPTIONS• Hurt conversion rate• Reduce effectiveness
The Evolution of A/B Testing
@Nicole_Mintiens
MISCONCEPTIONS• Hurt conversion rate• Reduce effectiveness• Waste resources
The Evolution of A/B Testing
@Nicole_Mintiens
MISCONCEPTIONS• Hurt conversion rate• Reduce effectiveness• Waste resources• Damage ROI
The Evolution of A/B Testing
@Nicole_Mintiens
TESTING IS NOT
OPTIMIZATION
The Evolution of A/B Testing
@Nicole_Mintiens
TESTING IS NOT
OPTIMIZATION…but it plays a role.
The Evolution of A/B Testing
@Nicole_Mintiens
DON’T JUST TEST, OPTIMIZE.• Avoid mistakes
The Evolution of A/B Testing
@Nicole_Mintiens
DON’T JUST TEST, OPTIMIZE.• Avoid mistakes• Increase effectiveness
The Evolution of A/B Testing
@Nicole_Mintiens
DON’T JUST TEST, OPTIMIZE.• Avoid mistakes• Increase effectiveness• Improve ROI
The Evolution of A/B Testing
@Nicole_Mintiens
I’LL BE YOUR GUIDE• 7+ years optimizing multifamily
– PPC, SMM, SEO & CRO
The Evolution of A/B Testing
@Nicole_Mintiens
A/B TESTING BASICS• Statistical hypothesis-based
experimentation between variations of a control.
The Evolution of A/B Testing
@Nicole_Mintiens
A/B TESTING BASICS• Statistical hypothesis-based
experimentation between variations of a control.
• Goal: – Isolate differences that
maximize an outcome • i.e. conversion
The Evolution of A/B Testing
@Nicole_Mintiens
CONVERSION DEFINITIONAction YOU want the user to take.
The Evolution of A/B Testing
@Nicole_Mintiens
A/B/N TESTING
The Evolution of A/B Testing
@Nicole_Mintiens
A/B/N TESTING
The Evolution of A/B Testing
@Nicole_Mintiens
A/B/N TESTING
The Evolution of A/B Testing
@Nicole_Mintiens
• 2000 – Google runs first test to find optimum number of SERP
results.
A/B TESTING HISTORY
The Evolution of A/B Testing
@Nicole_Mintiens
• 2000 – Google runs first test to find optimum number of SERP
results.
• 2008 – A/B tests raise additional $60 million for Obama campaign.
A/B TESTING HISTORY
The Evolution of A/B Testing
@Nicole_Mintiens
• 2000 – Google runs first test to find optimum number of SERP
results.
• 2008 – A/B tests raise additional $60 million for Obama campaign.
• 2010– Obama campaign’s Director of Analytics founds Optimizely.
A/B TESTING HISTORY
The Evolution of A/B Testing
@Nicole_Mintiens
• 2000 – Google runs first test to find optimum number of SERP
results.
• 2008 – A/B tests raise additional $60 million for Obama campaign.
• 2010– Obama campaign’s Director of Analytics founds Optimizely.
• 2011 – Google has run over 7,000 A/B tests
A/B TESTING HISTORY
The Evolution of A/B Testing
@Nicole_Mintiens
MULTIFAMILY AVG CONVERSION RATE
Source: 2016 G5 Multifamily Google Analytics Data
The Evolution of A/B Testing
@Nicole_Mintiens
MULTIFAMILY AVG CONVERSION RATE
Source: 2016 G5 Multifamily Google Analytics Data
~2%The Evolution of A/B Testing
@Nicole_Mintiens
ECOMMERCE AVG CONVERSION RATE
~3%The Evolution of A/B Testing
@Nicole_Mintiens
AMAZON PRIME’S CONVERSION RATE
Source: https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time
The Evolution of A/B Testing
@Nicole_Mintiens
2366%Better than
average!
AMAZON PRIME’S CONVERSION RATE
Source: https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time
The Evolution of A/B Testing
@Nicole_Mintiens
AMAZON’S TECHNOLOGY
Source: Builtwith.com
The Evolution of A/B Testing
@Nicole_Mintiens
AMAZON’S TECHNOLOGY
Source: Builtwith.com
The Evolution of A/B Testing
@Nicole_Mintiens
SOFTWARE ADVANCEMENTS• Marketing automation
The Evolution of A/B Testing
@Nicole_Mintiens
SOFTWARE ADVANCEMENTS• Marketing automation • Easy to install
The Evolution of A/B Testing
@Nicole_Mintiens
SOFTWARE ADVANCEMENTS• Marketing automation • Easy to install• WYSIWYG editors
The Evolution of A/B Testing
@Nicole_Mintiens
WHAT A/B TESTING CAN DO
• Validate hypotheses
The Evolution of A/B Testing
@Nicole_Mintiens
WHAT A/B TESTING CAN DO
• Validate hypotheses• Inform choices
The Evolution of A/B Testing
@Nicole_Mintiens
WHAT A/B TESTING CAN DO
• Validate hypotheses• Inform choices • Insight for iteration
The Evolution of A/B Testing
@Nicole_Mintiens
WHAT A/B TESTING CAN’T DO• Determine YOUR
–Problem
Thanks, Boss!
The Evolution of A/B Testing
@Nicole_Mintiens
WHAT A/B TESTING CAN’T DO• Determine YOUR
–Problem –User’s
needs/wantsThanks, Boss!
The Evolution of A/B Testing
@Nicole_Mintiens
WHAT A/B TESTING CAN’T DO• Determine YOUR
–Problem –User’s
needs/wants
• Improve traffic–Quality
Thanks, Boss!
The Evolution of A/B Testing
@Nicole_Mintiens
WHAT A/B TESTING CAN’T DO• Determine YOUR
–Problem –User’s
needs/wants
• Improve traffic–Quality–Volume
Thanks, Boss!
The Evolution of A/B Testing
@Nicole_Mintiens
CAN DOES NOT EQUAL SHOULDTesting will waste your time without:• Enough traffic
The Evolution of A/B Testing
@Nicole_Mintiens
CAN DOES NOT EQUAL SHOULDTesting will waste your time without:• Enough traffic• Enough research
The Evolution of A/B Testing
@Nicole_Mintiens
CAN DOES NOT EQUAL SHOULDTesting will waste your time without:• Enough traffic• Enough research• Technical resources
The Evolution of A/B Testing
@Nicole_Mintiens
THEN HOW DO YOU IMPROVE?!
The Evolution of A/B Testing
@Nicole_Mintiens
A PROCESSto improve performance
informed by INSIGHTand defined by GOALS.
CONVERSION RATE OPTIMIZATION(CRO)
The Evolution of A/B Testing
@Nicole_Mintiens
A PROCESSto improve performance
informed by INSIGHTand defined by GOALS.
GOALS
CONVERSION RATE OPTIMIZATION(CRO)
The Evolution of A/B Testing
@Nicole_Mintiens
ALWAYS ROOM FOR IMPROVEMENT• Customer acquisition cost
The Evolution of A/B Testing
@Nicole_Mintiens
ALWAYS ROOM FOR IMPROVEMENT• Customer acquisition cost• Return on investment
The Evolution of A/B Testing
@Nicole_Mintiens
ALWAYS ROOM FOR IMPROVEMENT• Customer acquisition cost• Return on investment• Customer understanding
The Evolution of A/B Testing
@Nicole_Mintiens
ALWAYS ROOM FOR IMPROVEMENT• Customer acquisition cost• Return on investment• Customer understanding• Google’s perception
The Evolution of A/B Testing
@Nicole_Mintiens
GOALS & TRACKING
Source: http://borsen.dk/opinion/blogs/view/17/4159/den_svaereste_marketingdisciplin.html
The Evolution of A/B Testing
@Nicole_Mintiens
YOUR IDEAL RESIDENTS
Prospects
Valu
e
I only rent the most affordable apartments.
The Evolution of A/B Testing
@Nicole_Mintiens
YOUR IDEAL RESIDENTS
Prospects
Valu
e
I only rent the most affordable apartments.
I’ll rent near my tech coworkers regardless of price.
The Evolution of A/B Testing
@Nicole_Mintiens
CUSTOMER RESEARCH
Source: https://blog.kissmetrics.com/wp-content/uploads/2015/02/NYU.lowres.ifiwere.jpg
The Evolution of A/B Testing
@Nicole_Mintiens
“The EFFECTIVENESS of
any lead generation campaign
comes down to your HOOK.”Mike King @IPullRankFounder iPullRankConsultant to Fortune/INC500 brands.
The Evolution of A/B Testing
@Nicole_Mintiens
VOICE OF CUSTOMER RESEARCH• What makes your
property unique?
The Evolution of A/B Testing
@Nicole_Mintiens
VOICE OF CUSTOMER RESEARCH• What makes your
property unique?
• Why you and not the competition?
The Evolution of A/B Testing
@Nicole_Mintiens
VOICE OF CUSTOMER RESEARCH• What makes your
property unique?
• Why you and not the competition?
The Evolution of A/B Testing
@Nicole_Mintiens
UNIQUE VALUE PROPOSITION (UVP):Above-the-fold EXPERIENCEthat communicates WHATyou do and WHYit’s UNIQUE.
The Evolution of A/B Testing
@Nicole_Mintiens
Original (A)
UVP: FIVE SECOND TEST
The Evolution of A/B Testing
@Nicole_Mintiens
Original (A) Variation (B)
UVP: FIVE SECOND TEST
The Evolution of A/B Testing
@Nicole_Mintiens
Original (A) Variation (B)
UVP: FIVE SECOND TEST
The Evolution of A/B Testing
@Nicole_Mintiens
CUSTOMER SEGMENTATION
The Evolution of A/B Testing
@Nicole_Mintiens
CUSTOMER SEGMENTATION
User Persona Creator: https://xtensio.com/user-persona/
The Evolution of A/B Testing
@Nicole_Mintiens
Slide credit to Angie Schottmuller - Check out the full presentation:
http://www.slideshare.net/aschottmuller/content-marketing-onpage-optimization-holy-grail
The Evolution of A/B Testing
@Nicole_Mintiens
LARGEST LEAKS
Image source: https://s-media-cache-ak0.pinimg.com/564x/7e/c9/f9/7ec9f92ab4e7819c76011e432e760ece.jpg
The Evolution of A/B Testing
@Nicole_Mintiens
LARGEST LEAKS
Image source: https://s-media-cache-ak0.pinimg.com/564x/7e/c9/f9/7ec9f92ab4e7819c76011e432e760ece.jpg
The Evolution of A/B Testing
@Nicole_Mintiens
ANALYZE ENGAGEMENT PATTERNSHeatmaps
The Evolution of A/B Testing
@Nicole_Mintiens
ANALYZE ENGAGEMENT PATTERNSHeatmaps
The Evolution of A/B Testing
@Nicole_Mintiens
ANALYZE PERCEPTIONS
The Evolution of A/B Testing
@Nicole_Mintiens
ANALYSIS FRAMEWORKG5 D-LITE
Deliver Relevance - Match then exceed user expectations.D
L
I
T
EThe Evolution of A/B Testing
@Nicole_Mintiens
ANALYSIS FRAMEWORKG5 D-LITE
Lower Friction - Reduce user anxiety. Simplify their process.
Deliver Relevance - Match then exceed user expectations.D
L
I
T
EThe Evolution of A/B Testing
@Nicole_Mintiens
ANALYSIS FRAMEWORKG5 D-LITE
Improve Clarity - Ensure messaging is clear, concise & valued.
Lower Friction - Reduce user anxiety. Simplify their process.
Deliver Relevance - Match then exceed user expectations.D
L
I
T
EThe Evolution of A/B Testing
@Nicole_Mintiens
ANALYSIS FRAMEWORKG5 D-LITE
Improve Clarity - Ensure messaging is clear, concise & valued.
Lower Friction - Reduce user anxiety. Simplify their process.
Deliver Relevance - Match then exceed user expectations.
Trigger Action - Motivate conversion. Remove distraction.
D
L
I
T
EThe Evolution of A/B Testing
@Nicole_Mintiens
ANALYSIS FRAMEWORKG5 D-LITE
Improve Clarity - Ensure messaging is clear, concise & valued.
Lower Friction - Reduce user anxiety. Simplify their process.
Deliver Relevance - Match then exceed user expectations.
Experiment - Learn, test and optimize for your users.
Trigger Action - Motivate conversion. Remove distraction.
D
L
I
T
EThe Evolution of A/B Testing
@Nicole_Mintiens
“A solid test hypothesis is an
INFORMED SOLUTIONto a real problem - not an arbitrary guess. ”Michael Aagaard
@ContentVerveInternational Speaker and Senior CRO @Unbounce
The Evolution of A/B Testing
@Nicole_Mintiens
HYPOTHESIS TEMPLATEIf [cause], then [effect], because [rationale].
Source: https://help.optimizely.com/Ideate_and_Hypothesize/Design_a_hypothesis_that_drives_your_business_goals
The Evolution of A/B Testing
@Nicole_Mintiens
TESTING
The Evolution of A/B Testing
@Nicole_Mintiens
TESTING Wait… do you even have enough traffic?
The Evolution of A/B Testing
@Nicole_Mintiens
Gives you CONFIDENCE that the changes you make to
your website ACTUALLYhave a positive IMPACT.
https://www.optimizely.com/optimization-glossary/statistical-significance/
STATISTICAL SIGNIFICANCE
“”
The Evolution of A/B Testing
@Nicole_Mintiens
MULTIFAMILY AVG SESSIONS/DAY
Source: 2016 G5 Multifamily Google Analytics Data
The Evolution of A/B Testing
@Nicole_Mintiens
MULTIFAMILY AVG SESSIONS/DAY
Source: 2016 G5 Multifamily Google Analytics Data
66The Evolution of A/B Testing
@Nicole_Mintiens
TEST DURATION CALCULATOR
Source: https://www.convert.com/tools/ab-test-duration-calculator/
Image source: http://www.vintag.es/2016/09/the-good-old-days-12-crazy-vintage-ads.html
The Evolution of A/B Testing
@Nicole_Mintiens
TEST DURATION CALCULATOR
Source: https://www.convert.com/tools/ab-test-duration-calculator/
Image source: http://www.vintag.es/2016/09/the-good-old-days-12-crazy-vintage-ads.html
The Evolution of A/B Testing
@Nicole_Mintiens
LOW TRAFFIC OPTIMIZATION
Source: http://unbounce.com/landing-pages/i-dont-have-enough-traffic-to-ab-test-my-landing-pages/
The Evolution of A/B Testing
@Nicole_Mintiens
A/B TESTING TOOLS
The Evolution of A/B Testing
@Nicole_Mintiens
A/B TESTING TOOLS
The Evolution of A/B Testing
@Nicole_Mintiens
BUILD VARIATIONSUse technical resources to ensure proper setup and tracking
The Evolution of A/B Testing
@Nicole_Mintiens
BUILD VARIATIONSUse technical resources to ensure proper setup and tracking
The Evolution of A/B Testing
@Nicole_Mintiens
LAUNCH TESTOriginal
(A)Variation
(B)
The Evolution of A/B Testing
@Nicole_Mintiens
LAUNCH TESTOriginal
(A)Variation
(B)
The Evolution of A/B Testing
@Nicole_Mintiens
MEASURE RESULTS
Image source: https://www.optimizely.com/ab-testing/
The Evolution of A/B Testing
@Nicole_Mintiens
MEASURE RESULTS
Image source: https://www.optimizely.com/ab-testing/
The Evolution of A/B Testing
@Nicole_Mintiens
DRAW INSIGHTS AND ITERATE
The Evolution of A/B Testing
@Nicole_Mintiens
Your properties aren't 1-size fits all and your strategy shouldn't be either.
DRAW INSIGHTS AND ITERATE
The Evolution of A/B Testing
@Nicole_Mintiens
MOST IMPORTANTLYResearch and
user feedback is
VITAL.
The Evolution of A/B Testing
@Nicole_Mintiens
REPEAT
GOALS
The Evolution of A/B Testing
@Nicole_Mintiens
THERE IS NO
BEST COLOR
41 Shades of Blue
The Evolution of A/B Testing
@Nicole_Mintiens
INDUSTRY BENCHMARKS
• Contextless data
The Evolution of A/B Testing
@Nicole_Mintiens
INDUSTRY BENCHMARKS
• Contextless data• Weak conclusions
The Evolution of A/B Testing
@Nicole_Mintiens
INDUSTRY BENCHMARKS
• Contextless data• Weak conclusions• Blind duplication
The Evolution of A/B Testing
@Nicole_Mintiens
INDUSTRY BENCHMARKS
• Contextless data• Weak conclusions• Blind duplication • Tactics over strategy
The Evolution of A/B Testing
@Nicole_Mintiens
USE STRATEGY,NOT JUST TACTICS
• Avoid risks
The Evolution of A/B Testing
@Nicole_Mintiens
USE STRATEGY,NOT JUST TACTICS
• Avoid risks• Use data effectively
The Evolution of A/B Testing
@Nicole_Mintiens
USE STRATEGY,NOT JUST TACTICS
• Avoid risks• Use data effectively• Discover solutions to
YOUR problems
The Evolution of A/B Testing
@Nicole_Mintiens
FREEMIUM (OR CLOSE) RESOURCES• Research:
– Personas
– Tracking
– Heatmapping
– Test Duration Calculator
The Evolution of A/B Testing
@Nicole_Mintiens
FREEMIUM (OR CLOSE) RESOURCES• Research:
– Personas
– Tracking
– Heatmapping
– Test Duration Calculator
• Analysis Tools:
G5 D-LITE CRO Framework
The Evolution of A/B Testing
@Nicole_Mintiens
FREEMIUM (OR CLOSE) RESOURCES• Research:
– Personas
– Tracking
– Heatmapping
– Test Duration Calculator
• Analysis Tools:
G5 D-LITE CRO Framework
• Hypotheses:
If [cause], then [effect], because [rationale]
The Evolution of A/B Testing
@Nicole_Mintiens
FREEMIUM (OR CLOSE) RESOURCES• Research:
– Personas
– Tracking
– Heatmapping
– Test Duration Calculator
• Analysis Tools:
G5 D-LITE CRO Framework
• Hypotheses:
If [cause], then [effect], because [rationale]
• Testing:
The Evolution of A/B Testing
@Nicole_Mintiens
“CRO IS LEARNING WHYYOUR USERS aren’t
converting and FIXING IT.”The Evolution of A/B Testing
@Nicole_Mintiens
GET THESE SLIDES: goo.gl/TPUOgA
THANKS!
DON’T JUST TEST, OPTIMIZE.
The Evolution of A/B Testing
@Nicole_Mintiens
Recommended