Lars LofgrenMarketing Analyst - August 2012 [email protected] - Confidential - Do not distribute
SUMMER 2012 STATE OF A/B TESTING
@larslofgren #KISSwebinar
We’ll cover...
State of A/B Testing1
Q&A2
Demo3
Let’s start with the data!
WHO AREWe?
Who are we? 176 in total
0
15
30
45
60
Ecommerce SaaS Agency/Cons Publishing Mobile Other
What’s the size of our teams?
0
22.5
45
67.5
90
1-5 6-10 11-25 26-60 51-200 200+
What testing tools does everyone use?
0
22.5
45
67.5
90
Visual WebsiteOptimizer
GA ContentExperiments
Optimizely None Other
How many A/B tests did we run last month?
0
37.5
75
112.5
150
0 1-4 5-9 10-19 20+
It’s easy to get ahead of our competitors
How do we generate ideas for testing?
0
15
30
45
60
TestRandomly Experience Best Practices Research Needs
Step 1: Get Qualitative
Step 2: Make a prediction
Step 3: Confirm with an A/B test
How KISSmetrics generates testing ideas
Get qualitative insights1
Predict how to improve2
Confirm prediction with an A/B test3
OURResults
Do we regularly increase conversions?
0
20
40
60
80
Always Sometimes Rarely Never
The revenue we gain from A/B testing
0
17.5
35
52.5
70
None $1-1,000$1001-10,000
$10,001-100,000
No IdeaOver
$100,000
2 ways to make money with A/B testing
Option 1: Understand your customers first
Get qualitative insights1
Predict how to improve2
Confirm prediction with an A/B test3
You’ll find the BIG wins much faster
Option 2: Get relentless
How pros like Facebook and Twitter do it
Build an entire growth team1
Back-to-back testing2
Look for 100s of small improvements3
How do we A/B test with small amounts of data?
Do tests with customer interviews
Talk to people, give different options1
Look for trends2
Build and see if actions match3
Do we track results over the long term?
0
32.5
65
97.5
130
Yes No
The pitfalls of not tracking long term results
Variation B performs the best in this funnel
Variation A gives the most revenue
LET’S TRACK THE RETURN ONA/B Testing
You’ll need customer analytics that can track multiple purchases from the
same person.
First, make sure you can track lifetime value
Where do we find customer analytics?
OR
Second, connect A/B tools to customer analytics
What about Google Analytics?
Connects tests to goals, not ecommerce transactions.
Recommended
