A/B Puzzle

A/B PUZZLE How A/B tests lie to us, and

how to drive genuine improvement

Jason Cohen, founder & CTO @asmartbear http://wpengine.com

The puzzle

758 out of 24,483 visits 85% confidence

What do stats really tell us?

758 out of 24,483 visits 85% confidence

Finding the Cheaters

105 athletes. 100 are clean. 5 are juicing. Test is 95% accurate.

Finding the CheatersIN REALITY

Clean (100) Juicing (5)

TEST RESULT (95%accuracy)

Clean

Juicing 5

095

5

Insight: Rare events

are indistinguishable from random noise

758 out of 24,483 visits 85% confidence

Easy significance testing with hamsters

Trust in the Hamster.

bit.ly/abhamster

Easy significance testing

N = A+B = total # of conversions.

D = (A–B)/2 = half the difference.

Significant if D2 > N.

bit.ly/abhamster

N = 758 + 685 = 1443

D = ( 758 — 685 ) / 2 = 37

1369 > 1443?

No! Not significant.

Final Result: As predicted

Wasted another month

30

40

50

60

70

12/2/09 12/16/09 1/5/10 1/13/10 2/2/10 4/9/10 4/27/10 9/24/10 9/14/11 11/8/11

OtherInbox Splash Page

Trust the Hamster

Beware the Rare

Defeat rare by seeking only huge results

Can you A/B test your way to huge results?

1,116

250

4,100

1,151Individuals: 67% off

Group: 3% off

Wisdom of the Crowd

“I’m as clueless as you, but in the equal and opposite direction.”

What’s the funniest joke?

And the winner is…

Jokes, Pizza, Design, Features, Voice,

Values, Culture

Crowds find objective truth but kill creativity

Beware the rare. Seek huge results.

Don’t let data kill creativity.

Jason Cohen, founder & CTO @asmartbear http://wpengine.com