Secrets from the frontlines - Content Jam · 2018. 10. 16. · 11 PDP Returning visitor comparing...

Secrets from the frontlines:

@ayat

AB test your way to better conversions

About meQueen of CRO for past 12 years

Move over Mother of Dragons, because I am the mother of not 3, but 4 monsters

Youngest is a mini Elmyra

If I find downtime: movie and TV show buff

@ayat

In case youforgotELMYRA

@ayat

is not easyCRO

@ayat

Starts with excitement and seems EASY

Then things start looking FUNNY(like each other)

Things get complicated so youGIVE UP

A case study with uplift from one A/B test529%

A case study with uplift from one A/B test86%

Google

10%

VWO

12%

Optimizely

25%

Successful A/B tests

Most of thesetests areOUTLIERS

@ayat

@ayat

“If you can increase the number of experiments you try from a hundred to a thousand, you dramatically increase the number of innovations you produce.”

- Amazon CEO Jeff Bezos

@ayat

IT ALWAYS COMES DOWN TO THE CUSTOMER

PE

RF

OR

MA

NC

E

TIME

The age of instant gratification

The naturalCycle of change

Future State

What we expect

Current State

@ayat

WE ARE IN THE AGE BEYOND COMMODITIZEDPRODUCTS AND SERVICES

Testing

FixingUI/UX

Collaboratingwith teams

Innovating

Gettingthe big win

DELIGHT: visitors by :

Lesson 1It’s not about moving pixels 1

CROis not changingthe color of a CTA

@ayat

Stop lying to yourself

There are no best practices

There is no checklist

You won’t get anywhere just tweaking

@ayat

Best practices often fail, as well. You are looking for repeatable,

sustainable growth.””

@ayat

Qualitative research

Quantitative research

Prioritize Hypothesis Test Post test analysis

Typical CRO Process

@ayat

0.90%

1.10%

1.30%

1.50%

1.70%

0 1 2 3 4 5 6 7 8

Conversion Rate

Testing That Generates a 5% Uplift

0.90%

1.10%

1.30%

1.50%

1.70%

0 1 2 3 4 5 6 7 8

Conversion Rate

Testing That Generates a 7% Uplift

0.90%

1.10%

1.30%

1.50%

1.70%

0 1 2 3 4 5 6 7 8

Conversion Rate

Testing That Generates a 10% Uplift

Lesson 2Beyond all males & females over 202

How do you fix the problem of long wait time at the elevator ?

I think or feel it but don’t or won’t say it

Top of Mind

I sense it but can’t find words to express it

I’m not aware of it but might recognize it if promoted

I can’t recognize it but it strongly influences how I feel and what I do

CONSCIOUS

SUBC

ONSCIOUS

Traditional “voice of the customer” methods get at these..

… and some of these

How we think visitors behave

@ayat

How visitors really behave

@ayat

@ayat

A different kind of customer journey

@ayat

@ayat

It always comes down to one thing

@ayat

An experience of a lifetime

PLANNING THE RESEARCHyou determine the research goal, what questions to ask, what methods to use,

how many people will participate.

CONDUCTING THE RESEARCHyou carry out your research

and gather data.

ANALYZING THE RESULTSyou analyze the collected data looking

for patterns and actionable insights.

Steps of qualitative research

One-on-One Focus Group Usability Surveys

Type of Knowledge

General Data about market

Focus on a narrow website functionality

Ability to interact with participants

In depth persona views

General views about market

No market knowledge

Limited market knowledge

Maybe

Maybe

Qualitative research methods overview

STEPS TO QUAL RESEARCH

Identify if you even need it (to solve current problem)

Identify your goal

1 2

Identify your personal bias Identify your participants

4 3

Don’t limit your research methods Format questions correctly

5 6

Analyze data and propagate

7

1. Can’t find the information I need

2. Don’t know if the item fits my phone

3. Want a different color

4. Can’t find the shipping costs

5. Comparing prices

Result: understanding of what bothers the visitors + testing ideas

“WHAT IS PREVENTING YOU FROM MAKING A PURCHASE?”

Test Fix Right away Investigate Action

Can’t find the information I need

Don’t know if the item fits my phone

want a different color

Can’t find the shipping costs

Answers

want a different color

“WHAT IS PREVENTING YOU FROM MAKING A PURCHASE?”

WHY?

Social proof via reviews placement

Max of 4 benefits of the product displayed

More options for logical persona

Specific compatibility info clearly labeled

1. Can’t find the information I need

2. Not ready to sign-up

3. Feeling rushed

4. Looking for cost

5. What is it going to take?

Result: understanding of visitor concerns = testing ideas

“WHAT IS PREVENTING YOU FROM SIGNING UP?”

Test Fix Right away Investigate Action

Can’t find the information I need

Not ready to sign-up

Feeling Rushed

Looking for costs

Answers

What is it going to take?

“WHAT IS PREVENTING YOU FROM SIGNING UP?”

WHY?

Clarified Process

Clear benefits displayed

Price is clearly displayed

Free option clearly displayed

Two-page Sign-up

Who are you targeting?

People tell you who they are, but we ignore it –because we want them to be who we want them to be“ “

Not all customers are the sameNot all customers go through the same experience

When you optimize, you make it easierfor visitors to complete tasks…

• Every visitor comes to your site to:

• Answer a question

• Find a solution (product/service) to a problem

• Compare options

• Convert

• Complete a task

• What task are they trying to accomplish?

• What behavior you’re trying to help and facilitate?

Seven Questions To Ask

1. What is the visitor intent?

2. What is the context of the visit?

3. Where does the experience start?

4. Where does the experience finish?

5. What are your visitor feeling entering the experience?

6. How brand-aware is the visitor?

7. Where is the visitor in the buying funnel?

@ayat

CUSTOMER EXPERIENCES

Experience Intent Context Start EndVisitor feeling

upon entering

Brand-

awarenessBuying stage

User searching for skin care

products for men on google

and lands on website

Wondering if

there is a good

skin care product

Sitting at home,

browsing on

mobile phone

Landed on the

homepageCart page

Looking around –

low level of

interest

Non-recognition Early

User on our email list gets a

promotional email with a

10% discount

Looking to solve

an existing

problem

Clicked on an

email offer using

mobile phone

Landed on

bookmarked page

Order

confirmationExcited

Brand

RecognitionLate

User watches video of

influencers showing the

effectiveness of our product

Wondering if

there is a good

solution to his

skin problems

Clicked on an ad

in an influencer

youtube channel

Landed on

custom landing

page

First step of

checkout

Curious –

excitedNon-recognition Mid

Lesson 3Think outside the box 3

HISTORY OF OPTIMIZATION

Create websites for traffic

Optimize for online transactions

Create responsive sites on mobile

We never just optimized for mobile.

FROM DESKTOP

@ayat

EBAY APP OVER THE YEARS

@ayat

Source: https://www.youtube.com/watch?v=xiyOTcVHqHwMinute : 36.11

Ten Years of eBay Android App Design

@ayat

Source: https://www.statista.com/statistics/249863/us-mobile-retail-commerce-sales-as-percentage-of-e-commerce-sales/

U.S. mobile retail commerce sales as percentage of retail e-commerce sales from 2017 to 2021

The latest trend now: MOBILE FIRST

FROM MOBILE

@ayat

Well the Chinese knew this already

Lesson 4Where to start? 4

Typical approach to making changes on a site

Because we saw (HIPPO hates this)

We expect that (removing the new filtration design that performs better

than original) will cause(the office to cool down for a

day)

We’ll measure this using(by fudging the numbers, and lying about the truth)

It should never be about putting out fires

ALL MARKETING ACTIVITIESSHOULD HAVE A PROCESS

Image source: https://gph.is/1R3fsmm

MOST COMPANIES BELIEVE IN IT, BUT MOST OF THEM DO NOT KNOW HOW TO DO IT

Image source: http://gph.is/2ezw1jJ

What are the results of the scrutinize phase?

Page Experience Item Discovered Type

1 HP New visitor looking to purchase Use of stock image that does Expert review

2 HP New visitor looking to purchase Lack of continuity in Expert review

3 HP New visitor looking to purchase Too many competing paths… Qual. analysis

4 HP New visitor looking to purchase Competing calls to action… Qual. analysis

5 PDP Returning visitor comparing No clear value proposition… Qual. analysis

6 PDP Returning visitor comparing CSS styling issues on Edge Quant. analysis

7 PDP Returning visitor comparing Lack of search functionality… Quant. analysis

8 PDP Returning visitor comparing CSS styling issues on Edge Quant. analysis

9 PDP Returning visitor comparing Lack of search functionality… Quant. analysis

10 PDP Returning visitor comparing CSS styling issues on Edge Quant. analysis

11 PDP Returning visitor comparing Lack of search functionality… Quant. analysis

12 CDP New visitor looking to purchase CSS styling issues on Edge Quant. analysis

13 CDP New visitor looking to purchase Lack of search functionality… Expert review

14 CDP New visitor looking to purchase CSS styling issues on Edge Quant. analysis

15 CDP New visitor looking to purchase Lack of search functionality… Expert review

Identify & Classify

F R A IT RO IF

FIX IT RIGHT AWAY(stop the bleeding)

INSTRUMENT RESEARCH OPPORTUNIT

Y

INVESTIGATE FURTHER

F R A IT RO IF

Where should I start?

• Separate problem identification from possible solutions

• Use 18 different factors to weight each line item

• Assign different priorities for different line items

INVESP prioritization model

• Potential Impact

• Was the problem discovered through expert review?

• Was the problem discovered through qualitative research?

• Was the problem discovered through user testing?

• Was the problem discovered through heatmaps/Video recording?

• Was the problem discovered through analytics?

• Problem above the fold?

OBJECTIVE

SUBJECTIVEPERSONALOPINION

DATADRIVEN

Invesp prioritization model - Problem factors

Does the test idea enhance primary conversion goal?

Is implementing the change easy (politically, LOE)?

Adding or removing an element?

Does the test idea increase visitor trust?

Does the test idea reduce FUDs?

Does the test idea incentivize visitors to act right away?

Does the test idea increase engagement?

Type of change

Type of page

% of page views

Invesp prioritization model – Testing ideas

Page Experience Item Discovered Type Classification Priority

1 HP New visitor looking to purchase Use of stock image that does Expert review Stop the bleeding 11

2 HP New visitor looking to purchase Lack of continuity in Expert review Research opp. 12

3 HP New visitor looking to purchase Too many competing paths… Qual. analysis Instrument 13

4 HP New visitor looking to purchase Competing calls to action… Qual. analysis Research opp. 1

5 PDP Returning visitor comparing No clear value proposition… Qual. analysis Research opp. 14

6 PDP Returning visitor comparing CSS styling issues on Edge Quant. analysis Research opp. 2

7 PDP Returning visitor comparing Lack of search functionality… Quant. analysis Investigate Further 15

8 PDP Returning visitor comparing CSS styling issues on Edge Quant. analysis Stop the bleeding 3

9 PDP Returning visitor comparing Lack of search functionality… Quant. analysis Investigate Further 4

10 PDP Returning visitor comparing CSS styling issues on Edge Quant. analysis Investigate Further 5

11 PDP Returning visitor comparing Lack of search functionality… Quant. analysis Stop the bleeding 8

12 CDP New visitor looking to purchase CSS styling issues on Edge Quant. analysis Research opp. 6

13 CDP New visitor looking to purchase Lack of search functionality… Expert review Research opp. 7

14 CDP New visitor looking to purchase CSS styling issues on Edge Quant. analysis Research opp. 9

15 CDP New visitor looking to purchase Lack of search functionality… Expert review Test 10

ClassificationResearch opportunityInvestigate furtherStop the bleedingInstrument

ProblemsNew Features

Type

Post prioritization

Lesson 5Threat awareness 5

Scientificmethod

Source: https://en.wikipedia.org/wiki/Scientific_method

Type I and Type II Errors

Because we saw (data/feedback)

Source: bit.ly/hyp_kit

We expect that (change) will cause (impact)

We’ll measure this using (data metric)

Sample hypothesis

Because we observed visitors high cart abandonment and customer interviews and polls confirmed concerns for free shipping

We believe that exposing free delivery earlier and repeatedly

With all visitors Will cause a reduction in cart abandonment and increase conversions by 5%.

We will measure via AB test and post test poll.Source: OptimizeorDie

Hypothesis

Sample hypothesis

Sample hypothesis

1. What is the current observed behavior in terms of metrics?

• 25% of all visitors click on the add to cart button

• 10% of returning visitors fill the contact form

2. Critical to every hypothesis is the potential impact you think your change will produce (is it a 5%, 10%, 20% uplift?)

• By introducing the new design, we will increase the click on the add to cart button to 30% for all visitors

• By introducing the new design, 12% of our returning visitors will fill the contact form

Measure and set aggressive goals

“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…”

- Amazon CEO Jeff Bezos

@ayat

@ayat

A SPECIAL GIFT

Your Complete ConversionOptimization Start-up Kit

Download Your CRO Start-up Kit:http://bit.ly/startcro

THANK YOU!

Stay connected

@ayat in/ayatshukairy/ayat@invesp.com