YOU SHOULD TEST THAT!

Kelley Howell, UX Product Manager

Why should we test? How we think users will act is often

different than how they really act It is the best way to settle disputes

about designs

Why should we test?

Testing provides a direct way for us to identify users and their interactions with the website

Other tools to find out how we’re doing Usability Testing Field Studies Eye Tracking Click Mapping Multivariate Testing A/B Testing

Click icon to add picture

What is A/B Testing?

A/B Testing is…• Two versions of a page are created• Traffic is split between versions• Users have a 50% chance of seeing

one of the versions

It is important that only one element on a page is changed. Any more than that creates a multivariate test which has a different set of testing parameters.

Click icon to add picture

Advantages of A/B Testing

Easy to create and implement tests

Flexibility in choosing testing objectives

Can be adapted for any traffic numbers

Measures actual behavior of users

Popular A/B Testing tools

Click icon to add picture

How to A/B Test

Before you start testing• Prioritize pages• Define website goals• Create an hypothesis of what you

expect to occur• Define which metrics you are going to

measure and what those metrics are going to measured against

Decide which one thing you want to test

Know what variable you are testing for

Determine which variable to test

Redacted Create alternatives

Create Alternatives

Redacted

Devote time to let results mature

Redacted

Test, test, and keep testing!

Click icon to add picture

What Can you A/B Test?

Call to Actions

Microcopy

Page Layout

Form Length

A/B testing can’t answer everything

Click icon to add picture

Famous A/B Tests

Fact: A simple change was worth 300 Million Dollars

Fact: Google infamously tested 41 different shades of blue

Fact: This A/B test made A/B testing cool

Click icon to add picture

Things that can ruin your A/B test

Selecting Certain Population Sources

Ignoring Temporal Effects

Stopping Your Test Early

Click icon to add picture

Disadvantages in A/B Testing

1) Results change as time marches forward

A/B testing: one at a time

At a certain level of maturity, dramatic results usually require dramatic design tests

4) Not all A/B Testing Goals are testable

Click icon to add picture

A/B Testing in a nutshell

Define your goals Make up a hypothesis to test Create a variation of your original Set goals on what to fine Run you test for long period Test, and keep testing to

see if result holds true

That’s a Wrap!

Remember: Testing takes time, don’t expect overnight miracles– and if that happens something’s wrong!

For more research-based insights, check out the UX insights portal: http://redacted.com

THOUGHTS? QUESTIONS?