WTW TLE 2015 Powerpoint_Hua_Charles-final

© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved. #TLE2015

How To Win Friends & Influence People With A/B Testing

@

Charles Hua Director of Optimization & AB Testing


Real-Life Test: Which is True About Charles?

I have 2 kids:(older kid’s birthday

month/day/year) x 2 =(younger kid’s birthday

month/day/year)

Version A

At one point, I considered a career as an accountant

Version B

How good is your gut?


Web Analyst

Project Manager

QA

UX Designer

Tagging Eng

Frontend Developer

Statistician

Politician

Mom

Experiment Designer

© 2015, WhichTestWon, a division of Anne Holland Ventures Inc. All rights reserved.

I perform experiments on people…

#TLE2015


What the heck is RetailMeNot ?

#TLE2015

Multinational coupon company with the largest coupon sites in US and UK

We are here to save the world… some money!


2013 2014

30100

700+

Testing VolumeVeg CQMS


+12M Annualized incremental revenue with just one algorithm test

#TLE2015


Making sound business decisions with DATA

What Is The Purpose Of Testing @RMN?

#TLE2015


Build a solid foundation for success with Tools Process and People

#TLE2015

Get your House in order

1. Involve Others

Know what They are thinking


Step 1: Get The House In Order

#TLE2015


Right Tools For The Right Job

#TLE2015

Testing Tool Project Management


If You Don’t Do These, You Are Heading For Trouble

#TLE2015


Calibrate The Testing Tool Regularly

#TLE2015

AA or Blank Test


Filter Out Bots & Other Traffic Noise

#TLE2015

Keep the IP filter list up to date


Create A Framework For Outliers Then Monitor Them

#TLE2015

3 Standard Deviations


Check page load speed

#TLE2015

100mS increase in load time ~2% increase in bounce


Create a Process

#TLE2015

1 Define roles and responsibility

2 Set scope and cadence

3 Create test threshold

4 Know when to stop a test

5 Retrospective


Set Expectations w/ Test Spec

#TLE2015

1 Hypothesis

2 Decision KPI

3 Sample size needed

4 Experiment design

5 What to do with win?


Importance Of Both Quantitative And Qualitative Kpis

#TLE2015

-18% in NPS due to Drop in Promoter


How To Monitor A Test In Flight?

#TLE2015

1. Create a Threshold2. Don’t panic


Do You Have The Confidence & Power?

#TLE2015

90% Confident

80% Power


Why Not Peek Before Hitting Sample Size?

#TLE2015

Control Test

Order 1 2

Visits 5 5

Conversion 20% 40%

Lift 100%

Confidence 90%

Power 16%

64 samples needed


How Is It Trending?

#TLE2015

Look for diverging or parallel lines


Congrats! 15% of your work is done.

Now you have to deal with PEOPLE…

#TLE2015


Step 2: Involve Others

#TLE2015


Testing

Product

Eng

UX / Design

Ops / MerchFinance

Analyst

SEO

Informed Decision Making

Form Core Testing A-Team & Educate The Process


LIFT Brainstorm Sessions

#TLE2015

Structured and Focused


Turn Ideas Into Experiment Design

#TLE2015

Combine ideas into MVT

Element 1 Element 2

Variation 1 Variation 1



2 X 3 = 6 Variations


Build An Efficient Roadmap

#TLE2015

1 EVFPIE (Potential Importance Ease)

2 Room for the unexpected and HiPPOs


Be Transparent

#TLE2015

Communicate

Communicate

Communicate


Over-communicate, Share & Be Transparent

#TLE2015

WIKIEmail

Notifications


Discuss the results before making decisions…

#TLE2015


Promoting A Testing Culture – Make It Fun

#TLE2015


Step 3: Know What They Are

Thinking

#TLE2015


Who are they and what are they thinking?

Decision Makers?

#TLE2015


Questions To Ask

#TLE2015

1 What is the purpose of testing?

2 Who are the supporters?

3 How is the incentive structure set up?

4 Where is the testing team in the org?


Common Reasons People Push Back On Testing

#TLE2015

It slows me down

I’m afraid to know

I’m confident it’s

a win


Prep The Sirens

#TLE2015


Homepage Test

#TLE2015

Merchandising Team (Offers)

Design Team (Aesthetics, UX)

Sales Team($$$)

Operations Team(No Issues)

Marketing Team(Traffic)


Measure ROI

#TLE2015

1 Make friends at Finance Dept

2 Define measurement framework

3 Set up goals

4 Show sequential performance


Show them you care and make them feel important

#TLE2015


What’s HOT?

Get your HOUSE in order

Involve OTHERS

Know what THEY are thinking

#TLE2015


Continuous Innovation

#TLE2015


Sequential Likelihood Ratio Test – Bayesian Stats To complement Student T-Test Stats

#TLE2015

Test End (T-Test)Test End (SS)


Flexible Testing Platform

#TLE2015

Easy Integration With Any Testing Tools


Questions?

#TLE2015


Charles HuaRetailMeNotDirector of Optimization & AB [email protected]


Real-Life Test: Which is True About Charles?

I have 2 kids:(older kid’s birthday

month/day/year) x 2 =

(younger kid’s birthdaymonth/day/year)

Version A

At one point, I considered a career as an accountant

Version B

How good is your gut?

Documents

WTW TLE 2015 Powerpoint_Hua_Charles-final