Embed Size (px)
Citation preview
Secrets from the frontlines:
@ayat
AB test your way to better conversions
About meQueen of CRO for past 12 years
Move over Mother of Dragons, because I am the mother of not 3, but 4 monsters
Youngest is a mini Elmyra
If I find downtime: movie and TV show buff
@ayat
In case youforgotELMYRA
@ayat
is not easyCRO
@ayat
Starts with excitement and seems EASY
Then things start looking FUNNY(like each other)
Things get complicated so youGIVE UP
A case study with uplift from one A/B test529%
A case study with uplift from one A/B test86%
10%
VWO
12%
Optimizely
25%
Successful A/B tests
Most of thesetests areOUTLIERS
@ayat
@ayat
“If you can increase the number of experiments you try from a hundred to a thousand, you dramatically increase the number of innovations you produce.”
- Amazon CEO Jeff Bezos
@ayat
IT ALWAYS COMES DOWN TO THE CUSTOMER
PE
RF
OR
MA
NC
E
TIME
The age of instant gratification
The naturalCycle of change
Future State
What we expect
Current State
@ayat
WE ARE IN THE AGE BEYOND COMMODITIZEDPRODUCTS AND SERVICES
Testing
FixingUI/UX
Collaboratingwith teams
Innovating
Gettingthe big win
DELIGHT: visitors by :
Lesson 1It’s not about moving pixels 1
CROis not changingthe color of a CTA
@ayat
Stop lying to yourself
There are no best practices
There is no checklist
You won’t get anywhere just tweaking
@ayat
Best practices often fail, as well. You are looking for repeatable,
sustainable growth.””
@ayat
Qualitative research
Quantitative research
Prioritize Hypothesis Test Post test analysis
Typical CRO Process
@ayat
0.90%
1.10%
1.30%
1.50%
1.70%
0 1 2 3 4 5 6 7 8
Conversion Rate
Testing That Generates a 5% Uplift
0.90%
1.10%
1.30%
1.50%
1.70%
0 1 2 3 4 5 6 7 8
Conversion Rate
Testing That Generates a 7% Uplift
0.90%
1.10%
1.30%
1.50%
1.70%
0 1 2 3 4 5 6 7 8
Conversion Rate
Testing That Generates a 10% Uplift
Lesson 2Beyond all males & females over 202
How do you fix the problem of long wait time at the elevator ?
I think or feel it but don’t or won’t say it
Top of Mind
I sense it but can’t find words to express it
I’m not aware of it but might recognize it if promoted
I can’t recognize it but it strongly influences how I feel and what I do
CONSCIOUS
SUBC
ONSCIOUS
Traditional “voice of the customer” methods get at these..
… and some of these
How we think visitors behave
@ayat
How visitors really behave
@ayat
@ayat
A different kind of customer journey
@ayat
@ayat
It always comes down to one thing
@ayat
An experience of a lifetime
PLANNING THE RESEARCHyou determine the research goal, what questions to ask, what methods to use,
how many people will participate.
CONDUCTING THE RESEARCHyou carry out your research
and gather data.
ANALYZING THE RESULTSyou analyze the collected data looking
for patterns and actionable insights.
Steps of qualitative research
One-on-One Focus Group Usability Surveys
Type of Knowledge
General Data about market
Focus on a narrow website functionality
Ability to interact with participants
In depth persona views
General views about market
No market knowledge
Limited market knowledge
Maybe
Maybe
Qualitative research methods overview
STEPS TO QUAL RESEARCH
Identify if you even need it (to solve current problem)
Identify your goal
1 2
Identify your personal bias Identify your participants
4 3
Don’t limit your research methods Format questions correctly
5 6
Analyze data and propagate
7
1. Can’t find the information I need
2. Don’t know if the item fits my phone
3. Want a different color
4. Can’t find the shipping costs
5. Comparing prices
Result: understanding of what bothers the visitors + testing ideas
“WHAT IS PREVENTING YOU FROM MAKING A PURCHASE?”
Test Fix Right away Investigate Action
Can’t find the information I need
Don’t know if the item fits my phone
want a different color
Can’t find the shipping costs
Answers
want a different color
“WHAT IS PREVENTING YOU FROM MAKING A PURCHASE?”
WHY?
Social proof via reviews placement
Max of 4 benefits of the product displayed
More options for logical persona
Specific compatibility info clearly labeled
1. Can’t find the information I need
2. Not ready to sign-up
3. Feeling rushed
4. Looking for cost
5. What is it going to take?
Result: understanding of visitor concerns = testing ideas
“WHAT IS PREVENTING YOU FROM SIGNING UP?”
Test Fix Right away Investigate Action
Can’t find the information I need
Not ready to sign-up
Feeling Rushed
Looking for costs
Answers
What is it going to take?
“WHAT IS PREVENTING YOU FROM SIGNING UP?”
WHY?
Clarified Process
Clear benefits displayed
Price is clearly displayed
Free option clearly displayed
Two-page Sign-up
Who are you targeting?
People tell you who they are, but we ignore it –because we want them to be who we want them to be“ “
Not all customers are the sameNot all customers go through the same experience
When you optimize, you make it easierfor visitors to complete tasks…
• Every visitor comes to your site to:
• Answer a question
• Find a solution (product/service) to a problem
• Compare options
• Convert
• Complete a task
• What task are they trying to accomplish?
• What behavior you’re trying to help and facilitate?
Seven Questions To Ask
1. What is the visitor intent?
2. What is the context of the visit?
3. Where does the experience start?
4. Where does the experience finish?
5. What are your visitor feeling entering the experience?
6. How brand-aware is the visitor?
7. Where is the visitor in the buying funnel?
@ayat
CUSTOMER EXPERIENCES
Experience Intent Context Start EndVisitor feeling
upon entering
Brand-
awarenessBuying stage
User searching for skin care
products for men on google
and lands on website
Wondering if
there is a good
skin care product
Sitting at home,
browsing on
mobile phone
Landed on the
homepageCart page
Looking around –
low level of
interest
Non-recognition Early
User on our email list gets a
promotional email with a
10% discount
Looking to solve
an existing
problem
Clicked on an
email offer using
mobile phone
Landed on
bookmarked page
Order
confirmationExcited
Brand
RecognitionLate
User watches video of
influencers showing the
effectiveness of our product
Wondering if
there is a good
solution to his
skin problems
Clicked on an ad
in an influencer
youtube channel
Landed on
custom landing
page
First step of
checkout
Curious –
excitedNon-recognition Mid
Lesson 3Think outside the box 3
HISTORY OF OPTIMIZATION
Create websites for traffic
Optimize for online transactions
Create responsive sites on mobile
We never just optimized for mobile.
FROM DESKTOP
@ayat
EBAY APP OVER THE YEARS
@ayat
Source: https://www.youtube.com/watch?v=xiyOTcVHqHwMinute : 36.11
Ten Years of eBay Android App Design
@ayat
Source: https://www.statista.com/statistics/249863/us-mobile-retail-commerce-sales-as-percentage-of-e-commerce-sales/
U.S. mobile retail commerce sales as percentage of retail e-commerce sales from 2017 to 2021
The latest trend now: MOBILE FIRST
FROM MOBILE
@ayat
Well the Chinese knew this already
Lesson 4Where to start? 4
Typical approach to making changes on a site
Because we saw (HIPPO hates this)
We expect that (removing the new filtration design that performs better
than original) will cause(the office to cool down for a
day)
We’ll measure this using(by fudging the numbers, and lying about the truth)
It should never be about putting out fires
ALL MARKETING ACTIVITIESSHOULD HAVE A PROCESS
Image source: https://gph.is/1R3fsmm
MOST COMPANIES BELIEVE IN IT, BUT MOST OF THEM DO NOT KNOW HOW TO DO IT
Image source: http://gph.is/2ezw1jJ
What are the results of the scrutinize phase?
Page Experience Item Discovered Type
1 HP New visitor looking to purchase Use of stock image that does Expert review
2 HP New visitor looking to purchase Lack of continuity in Expert review
3 HP New visitor looking to purchase Too many competing paths… Qual. analysis
4 HP New visitor looking to purchase Competing calls to action… Qual. analysis
5 PDP Returning visitor comparing No clear value proposition… Qual. analysis
6 PDP Returning visitor comparing CSS styling issues on Edge Quant. analysis
7 PDP Returning visitor comparing Lack of search functionality… Quant. analysis
8 PDP Returning visitor comparing CSS styling issues on Edge Quant. analysis
9 PDP Returning visitor comparing Lack of search functionality… Quant. analysis
10 PDP Returning visitor comparing CSS styling issues on Edge Quant. analysis
11 PDP Returning visitor comparing Lack of search functionality… Quant. analysis
12 CDP New visitor looking to purchase CSS styling issues on Edge Quant. analysis
13 CDP New visitor looking to purchase Lack of search functionality… Expert review
14 CDP New visitor looking to purchase CSS styling issues on Edge Quant. analysis
15 CDP New visitor looking to purchase Lack of search functionality… Expert review
Identify & Classify
F R A IT RO IF
FIX IT RIGHT AWAY(stop the bleeding)
INSTRUMENT RESEARCH OPPORTUNIT
Y
INVESTIGATE FURTHER
F R A IT RO IF
Where should I start?
• Separate problem identification from possible solutions
• Use 18 different factors to weight each line item
• Assign different priorities for different line items
INVESP prioritization model
• Potential Impact
• Was the problem discovered through expert review?
• Was the problem discovered through qualitative research?
• Was the problem discovered through user testing?
• Was the problem discovered through heatmaps/Video recording?
• Was the problem discovered through analytics?
• Problem above the fold?
OBJECTIVE
SUBJECTIVEPERSONALOPINION
DATADRIVEN
Invesp prioritization model - Problem factors
Does the test idea enhance primary conversion goal?
Is implementing the change easy (politically, LOE)?
Adding or removing an element?
Does the test idea increase visitor trust?
Does the test idea reduce FUDs?
Does the test idea incentivize visitors to act right away?
Does the test idea increase engagement?
Type of change
Type of page
% of page views
Invesp prioritization model – Testing ideas
Page Experience Item Discovered Type Classification Priority
1 HP New visitor looking to purchase Use of stock image that does Expert review Stop the bleeding 11
2 HP New visitor looking to purchase Lack of continuity in Expert review Research opp. 12
3 HP New visitor looking to purchase Too many competing paths… Qual. analysis Instrument 13
4 HP New visitor looking to purchase Competing calls to action… Qual. analysis Research opp. 1
5 PDP Returning visitor comparing No clear value proposition… Qual. analysis Research opp. 14
6 PDP Returning visitor comparing CSS styling issues on Edge Quant. analysis Research opp. 2
7 PDP Returning visitor comparing Lack of search functionality… Quant. analysis Investigate Further 15
8 PDP Returning visitor comparing CSS styling issues on Edge Quant. analysis Stop the bleeding 3
9 PDP Returning visitor comparing Lack of search functionality… Quant. analysis Investigate Further 4
10 PDP Returning visitor comparing CSS styling issues on Edge Quant. analysis Investigate Further 5
11 PDP Returning visitor comparing Lack of search functionality… Quant. analysis Stop the bleeding 8
12 CDP New visitor looking to purchase CSS styling issues on Edge Quant. analysis Research opp. 6
13 CDP New visitor looking to purchase Lack of search functionality… Expert review Research opp. 7
14 CDP New visitor looking to purchase CSS styling issues on Edge Quant. analysis Research opp. 9
15 CDP New visitor looking to purchase Lack of search functionality… Expert review Test 10
ClassificationResearch opportunityInvestigate furtherStop the bleedingInstrument
ProblemsNew Features
Type
Post prioritization
Lesson 5Threat awareness 5
Scientificmethod
Source: https://en.wikipedia.org/wiki/Scientific_method
Type I and Type II Errors
Because we saw (data/feedback)
Source: bit.ly/hyp_kit
We expect that (change) will cause (impact)
We’ll measure this using (data metric)
Sample hypothesis
Because we observed visitors high cart abandonment and customer interviews and polls confirmed concerns for free shipping
We believe that exposing free delivery earlier and repeatedly
With all visitors Will cause a reduction in cart abandonment and increase conversions by 5%.
We will measure via AB test and post test poll.Source: OptimizeorDie
Hypothesis
Sample hypothesis
Sample hypothesis
1. What is the current observed behavior in terms of metrics?
• 25% of all visitors click on the add to cart button
• 10% of returning visitors fill the contact form
2. Critical to every hypothesis is the potential impact you think your change will produce (is it a 5%, 10%, 20% uplift?)
• By introducing the new design, we will increase the click on the add to cart button to 30% for all visitors
• By introducing the new design, 12% of our returning visitors will fill the contact form
Measure and set aggressive goals
“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…”
- Amazon CEO Jeff Bezos
@ayat
@ayat
A SPECIAL GIFT
Your Complete ConversionOptimization Start-up Kit
Download Your CRO Start-up Kit:http://bit.ly/startcro
