QA: How I Learned to Stop Breaking and Start Loving Tests

OPTIMIZATION QA: How to Avoid Launching Tests that Break Your SiteWhen you break shit you pollute data, lose money and eventually get fired. Here’s how to avoid that pain and look like a hero in the process.

@joeljharvey #convcon

Joel Harvey, COO Conversion Sciences

@joeljharvey #convcon

Goal

@joeljharvey #convcon

What’s QA?

@joeljharvey #convcon

Let’s say that …

@joeljharvey #convcon

30% of your visitors

get a broken experience on

your site

@joeljharvey #convcon

YOUR

SITE IS

BROKEN ON ONE OF

THESE

SCREENS

@joeljharvey #convcon

No One On Your Team Knows

@joeljharvey #convcon

20% RPVIE BUG

@joeljharvey #convcon

INTERNETEXPLORER

@joeljharvey #convcon

You’re about to hire this guy as your CPA

@joeljharvey #convcon

QA MATH

@joeljharvey #convcon

Internet Explorer Traffic: Feb 2013

Internet Explorer Traffic: Feb 2014

@joeljharvey #convcon

120% Increase Internet Explorer (IE) Revenue

20% Increase in All Revenue

20% Increase in IE Average Order Value

167% Increase in IE Conversion Rate

$1,500,000 Per Year

@joeljharvey #convcon

Who Are You?

@joeljharvey #convcon

Who Am I?- COO & Managing Partner,

Conversion Sciences- Conversion Scientist- Data Geek- Entrepreneurial MBA

@joeljharvey #convcon

1. RE-DEFINE QA

2. QA STRATEGY

3. QA TOOLS

4. QA PROCESS

RE-DEFINING QA

#1

@joeljharvey #convcon

STOP STEALING FROM YOURSELF

@joeljharvey #convcon

30% of your visitors

get a broken experience on

your site

QA DRIVES REVENUE

@joeljharvey #convcon

OLD BORING DEFINITION

@joeljharvey #convcon

@joeljharvey #convcon

QA is just a Cost Center

We can’t support everything!

QA prevents us from being agile

People on really old devices probably aren’t buyers

QA is an IT / Development Task

@joeljharvey #convcon

QA IS A MARKETING FUNCTION

@joeljharvey #convcon

QA STRATEGY

#2

@joeljharvey #convcon

INFUSE QA INTO YOUR OPTIMIZATION DNA

@joeljharvey #convcon

1. CULTURE

3. TECH PARADIGMS

2. REAL USER USE CASES

QA IS OPTIMIZATION

@joeljharvey #convcon

1. Understand your visitors

@joeljharvey #convcon

2. Make sure control is not broken

@joeljharvey #convcon

3. Generate great hypotheses & test them

@joeljharvey #convcon

@joeljharvey #convcon

4. Don’t pollute your test data

5. Do what the data tells you to do

@joeljharvey #convcon

BUILDING A QA CULTURE

@joeljharvey #convcon

@joeljharvey #convcon

#QAHAMMER

@joeljharvey #convcon

Build Destroy

HAMMER UP

@joeljharvey #convcon

BUY IN

@joeljharvey #convcon

@joeljharvey #convcon

@joeljharvey #convcon

20% RPVIE BUG

@joeljharvey #convcon

REAL USE CASES

@joeljharvey #convcon

90%

@joeljharvey #convcon

VISITORS DON’T USE

YOUR SITE THE WAY YOU

THINK THEY DO@joeljharvey #convcon

@joeljharvey #convcon

SCREEN RECORDING

@joeljharvey #convcon

USER TESTING

DOCUMENT AND QA ALL USE CASES

@joeljharvey #convcon

YOUR JOB IS TO BREAK YOUR

SITE

@joeljharvey #convcon

@joeljharvey #convcon

WHAT SHOULD YOU

SUPPORT?@joeljharvey #convcon

5%

@joeljharvey #convcon

@joeljharvey #convcon

Old New

BROWSER SUPPORT

Internet Explorer 8 Chrome 7 Firefox 4 Safari 4.1 iOS 5.1 *

Internet Explorer 11 Chrome 33 Firefox 27 Safari 7.0 iOS 7.1 *

@joeljharvey #convcon

OS SUPPORT

Old New

Windows XP OS X 10.4 (Tiger) Android 2.3 iOS 5.1 Windows Phone 7

Windows 8 OS X 10.9 (Maverick) Android 4.4 iOS 7.1 Windows Phone 8.1

5% MINIMUM

@joeljharvey #convcon

@joeljharvey #convcon

@joeljharvey #convcon

@joeljharvey #convcon

@joeljharvey #convcon

@joeljharvey #convcon

@joeljharvey #convcon

@joeljharvey #convcon

@joeljharvey #convcon

@joeljharvey #convcon

QA TOOLS

#4

@joeljharvey #convcon

BUILD YOUR QA’TION STATION

@joeljharvey #convcon

SEE YOUR SITES LIKE

YOUR VISITORS

iOS 7.1?

@joeljharvey #convcon

@joeljharvey #convcon

SOFTWAREvs.

HARDWARE

@joeljharvey #convcon

iPad, iOS 5

iPad, iOS 7.1

Windows Vista

Android 4.3

iPhones: iOS4, 5, 7 Android:

2.3.6, 4.0.4,

Windows 8, no

updates

OS X 10.6 Windows 8, regular updates

+ VM’s

OS X 10.8

OS X10.9

OS X 10.7 + VM’s

Windows XP

Windows 7

@joeljharvey #convcon

@joeljharvey #convcon

?

@joeljharvey #convcon

1. Browser

2. Operating System

3. “Other”

- Visitor Internet Speed

- Browser Plugins

- Flash

- Resolution

QA’TION STATION CHECKLIST

QA PROCESS

#4

@joeljharvey #convcon

BE THE HAMMER

QA EXPERIENCES,

NOT PAGES

@joeljharvey #convcon

@joeljharvey #convcon

@joeljharvey #convcon

@joeljharvey #convcon

TEST QA PROCESS

@joeljharvey #convcon

1.Build Your Test

@joeljharvey #convcon

2. Use Cookie, Query

String and/or Location Targeting so Only You

See Experience

@joeljharvey #convcon

3. Pause / Disable All

But One Experience

@joeljharvey #convcon

4. Review Live

Experience In All Supported

Browser/OS Combos (like REAL USERS)

@joeljharvey #convcon

5. Clear Cookies, Pause

Good-To-Go Experience and

Unpause the Next One

@joeljharvey #convcon

6. Repeat

@joeljharvey #convcon

A FEW TIPS

RANDOMIZE YOUR

APPROACH

CAUTION WITH CLASSES

WATCH SCRIPT STACKS

FEAR THE WYSIWYG

OWNERSHIP

@joeljharvey #convcon

Need to Borrow a QA Hammer?

Call 1-888-961-6604 and mention this session for a chance to win a Free QA Audit ($15,000 Value)

Conversion Conference San Francisco, 2014

Joel Harvey, Conversion Sciences

Joel@ConversionSciences.com