Google’s Website Optimizer

Overview

Part 1 – Guidelines for testing

• Pick the right pages to test

• Plan your tests

• Optimize and test again

Part 2 – Some sample tests and results

Tips and Lessons Learned

1. Pick the right pages to test

Identify your site goals

Find high volume / goal-oriented pages

• If your high-traffic pages don’t contribute to

your site goals, change them so they do

Use funnel analysis to see how traffic

flows to goal pages

2. Plan your tests

Predict what change will have the

greatest net effect and test that

Find ideas for testing from …

• Web pages you like

• Marketing studies, books, e-zine articles

• Experience with DM and EM

Keep it simple

What about low-traffic pages?

Consider statistical significance and time

to completion on low-traffic pages

Do tests on high-traffic pages and apply

lessons to low-traffic pages

Drive visitors to low-traffic pages during

the testing period

What to test

Headlines

Images and colors

Simple or complex page designs

Larger or smaller text

Longer or shorter pages

Offer / price

Call to action (color, wording, placement)

3. Optimize and test again

Test the winner

• If the long page beats the short page, you can

still improve the design of the long page

It never ends

• The more you test the more you’ll think of new

things to test

• Keep focused on the things that significantly

contribute to ROI

“Common Sense” vs. testing

The more you test the less you trust your

own opinion.

… or others’ opinion.

Before you follow “best practices,” make

sure they’re based on testing.

Google’s Website Optimizer

Overview

Part 1 – Guidelines for testing

Part 2 – Sample tests and results

Greg Krehbiel

gkrehbiel@kiplinger.com

Tips and Lessons Learned

1. Pick the right pages to test

• Identify your site goals

• Find high volume / goal-oriented pages

– If your high-traffic pages don’t contribute to

your site goals, change them so they do

• Use funnel analysis to see how traffic

flows to goal pages

2. Plan your tests

• Pick elements of the page that will have

the greatest net effect and test them

• Find ideas for testing from …

– Web pages you like

– Marketing studies, books, e-zine articles

– Experience with DM and EM

• Keep it simple

What about low-traffic pages?

• Consider statistical significance and time

to completion on low-traffic pages

• Do your tests on high-traffic pages and

apply results to low-traffic pages

• Drive visitors to low-traffic pages during

the testing period

What to test

• Offer / price

• Headlines

• Images and colors

• Simple or complex page / form designs

• Larger or smaller text

• Longer or shorter pages

• Call to action (color, wording, placement)

• Radically different page designs

Landing page Registration page “Accept order”

page

Test and Conversion Pages

Confirmation

page

This is your

test page

Which is the best conversion page?

Registration page as conversion page• Faster experiment

• Tests call to action on landing page

• Does not guarantee more orders

“Accept Order” page as conversion page• Tests landing page and registration page in combination

• Slower experiment

Confirmation page as conversion page• Slowest experiment

• Guarantees landing page success = more revenue

Test and Conversion Pages, 2

Use Funnel Analysis!

Whatever pages you use for

your test and conversion

pages, use funnel analysis

to track the entire process

from landing page to

conversion.

Google Analytics provides

funnel analysis.

3. Optimize and test again

• Test the winner

– If the long page beats the short page, you can

still improve the design of the long page

• Keep testing

– The more you test the more you’ll think of new

things to test

– Keep focused on the things that significantly

contribute to ROI

“Common Sense” vs. testing

• The more you test the less you trust your

own opinion.

• … or others’ opinion.

• Before you follow “best practices,” make

sure they’re based on testing.

Sample Tests and Results

What’s this guy doing here?

Why two order buttons?

Why three columns?

Is the text too small?

Is this the right headline?

Slight winner

Testing …

On the left, the

chart vs. the

image of the

newsletter

On the right, the

big messy image

vs. no image.

Breaking up the results

No image on the right hurts

Image of the newsletter vs. chart is not conclusive

Testing …

Two columns vs.

three columns

Two columns is

slightly better –

16% lift

Testing …

12 pt. vs. 14 pt.

type in the main

section of the

page.

14 pt. type is

slightly better –

12% lift.

Before and After

Standard

landing page

for the tax

letter

Testing a very different

page

Very long – at least three

screens on an average

browser

Includes …

• Text that worked in DM

• Testimonials

• Links to a sample issue

• Guarantee

It’s ugly, but it beat the control

7.5 to 1!

Adding the price near the

“order now” button

suppressed response in a

later test

Testing a prettier

version of the

long page

This prettier

version

suppressed

response by 5.2%

Why?

• Was it the blue

background?

• Was it because

the page was

centered?

Revised the

“pretty”

version

Moved it to the

left.

Changed the

background

color

The revised

version beat the

new control by

24%

Testing long copy for the

agriculture letter

This long, folksy letter from

the editor works well in DM

Page emphasizes “free

sample issue” offer, which

works well in our other

channels.

Simple page design

Beat the control by 106%

Remember that winning

page for The Kiplinger

Letter …

I decided to test this

against long copy as well.

Testing long

letter vs. short

page

This letter works

well in DM

Emphasizes

Kiplinger brand

Prominent “free

sample issue”

offer

Money-back

guarantee

Simple design

Long copy

But wait …

Why hide the

money-back

guarantee?

Moving the

guarantee to the

top of the page

is pulling an

added 10%

improvement

Testing 3 panels on e-

mail sign-up page

• Headline

• Special Report offer

• “Benefit” language

Lessons Learned

• Focus on … – What works in other channels

– Headlines

– Guarantee language

– Length of copy

– Complexity of the page

– Sample issue / images of the product

– Relevant images

• Don’t be satisfied with your results– Tweak winning and losing pages

– Repeat multi-panel tests to verify winners

– After refining a page, test it against something completely different