View
220
Download
0
Category
Preview:
Citation preview
Top 10 Tips for Mobile App Optimization using A/B Testing
Steve CollinsCTO / Cofounder Swrve
Top 10 Tips for Mobile App Optimization using A/B Testing
“One accurate measurement is worth 1,000 expert opinions.”
Admiral Grace Murray Hopper
Top 10 Tips for Mobile App Optimization using A/B Testing
A/B Testing
Split Test Measure Choose Deploy
Top 10 Tips for Mobile App Optimization using A/B Testing
Start simple
0101
Top 10 Tips for Mobile App Optimization using A/B Testing
#1: Start Simple
■Identify “low hanging fruit”■Choose minimal viable
implementation■Get a win as early as possible
Time to 1st interstitial? Tutorial guide
character?
vs
Force Login?
Top 10 Tips for Mobile App Optimization using A/B Testing
#1: Start Simple
Top 10 Tips for Mobile App Optimization using A/B Testing
#1: Start Simple
{variant = A}
UID = ABCD
Top 10 Tips for Mobile App Optimization using A/B Testing
#1: Start Simple
Event(Tut_A)
A: 1,356 completedB: 1,793 completed
Top 10 Tips for Mobile App Optimization using A/B Testing
Build a data driven culture
0202
Top 10 Tips for Mobile App Optimization using A/B Testing
■Data driven != lots of metrics / reports
#2: Data Driven Culture
Top 10 Tips for Mobile App Optimization using A/B Testing
#2: Data Driven Culture
■Beware the HiPPOs
Top 10 Tips for Mobile App Optimization using A/B Testing
#2: Data Driven Culture
■Link experiments to bottom line
Time to 1st interstitial?
CTR In-app Revenue
Top 10 Tips for Mobile App Optimization using A/B Testing
Remove latency
0303
Top 10 Tips for Mobile App Optimization using A/B Testing
#3: Remove Latency
■Close the loop
Measure
Take actionIterate, fail-fast
TEST
Top 10 Tips for Mobile App Optimization using A/B Testing
#3: Remove Latency
Measure / Adapt
Measure / Adapt
DevelopDevelopARTENG
PM
00111011100011100001000001010101
Test FrameworkTest Framework
Top 10 Tips for Mobile App Optimization using A/B Testing
Kill switch
0404
Top 10 Tips for Mobile App Optimization using A/B Testing
#4: Kill Switch
■3 Schools of thought:1. Fail early / fast
Top 10 Tips for Mobile App Optimization using A/B Testing
#4: Kill Switch
■3 Schools of thought:1. Fail early / fast2. Thorough QA
DraftDraft
StageStage
PublishPublish
QA Users
All Users
Top 10 Tips for Mobile App Optimization using A/B Testing
Top 10 Tips for Mobile App Optimization using A/B Testing
#4: Kill Switch■3 Schools of thought:
1. Fail early / fast2. Thorough QA3. Ramp-up
Top 10 Tips for Mobile App Optimization using A/B Testing
Minimize variation
0505
Top 10 Tips for Mobile App Optimization using A/B Testing
#5: Minimize Variation
A B
+102%
www.abtests.com and diythemes.com
Top 10 Tips for Mobile App Optimization using A/B Testing
Lookout for longitudinal impacts
0606
Top 10 Tips for Mobile App Optimization using A/B Testing
#6: Longitudinal Impacts
■Clear winning variant:
Conversions up
Top 10 Tips for Mobile App Optimization using A/B Testing
#6: Longitudinal Impacts
■But long-term KPIs are negative■Segment your metrics by variant
Test Start Time
KPI
Top 10 Tips for Mobile App Optimization using A/B Testing
#6: Longitudinal Impacts
■Beware temporal effects
24 hrs
Top 10 Tips for Mobile App Optimization using A/B Testing
Treat new and repeat users differently
0707
Top 10 Tips for Mobile App Optimization using A/B Testing
#7: Treat new and repeat users differently
A
B
Top 10 Tips for Mobile App Optimization using A/B Testing
#7: Treat new and repeat users differently
■“Primacy Effect”– Test new users only– Test users on a new app-version only– Ensure consistent user experience
Top 10 Tips for Mobile App Optimization using A/B Testing
Test offers not prices
0808
Top 10 Tips for Mobile App Optimization using A/B Testing
15 15
#8: Test offers not prices
■Sometimes better to avoid direct testing
20 20
Top 10 Tips for Mobile App Optimization using A/B Testing
2525
#8: Test offers not prices
■Sometimes better to avoid direct testing
25 25
-20%-40%
Top 10 Tips for Mobile App Optimization using A/B Testing
Run the math
0909
Top 10 Tips for Mobile App Optimization using A/B Testing
#9: Run the math
■Null hypothesis testing
Top 10 Tips for Mobile App Optimization using A/B Testing
#9: Run the math
■Null hypothesis testing
Top 10 Tips for Mobile App Optimization using A/B Testing
#9: Run the math
■How long to run test?
Top 10 Tips for Mobile App Optimization using A/B Testing
Save your results
1010
Top 10 Tips for Mobile App Optimization using A/B Testing
#10: Save your results
■ Name■ When it ran■ Owner■ Hypothesis■ Variants■ Result■ KPIs
Top 10 Tips for Mobile App Optimization using A/B Testing
Thanks
"Planet Cute" art by Daniel Cook (Lostgarden.com)48 px Web Icons by emey87.deviantart.com
■Some handy links:– http://www.swrve.com/blog/ – http://www.kaushik.net – http://20bits.com/article/statistical-analysis-and-ab-testing
– http://www.smashingmagazine.com/2010/06/24/the-ultimate-guide-to-a-b-testing/
Recommended