A/B Testing Pitfalls

Michal Parizek (@parezem), Avast #MeasureCamp , March 2015

Low-hanging Fruit

(1) 2 business cycles

(2) Big enough data sample (minimum of 200 orders per testing experience

(3) No bugs in A/B test setup

(4) Daily orders/revenue + cumulative orders/revenue

(5) Check http://abtestguide.com/calc/

@parezem | #MeasureCamp

Traffic Mix & Seasonality

Challenge

A/B test results are tight to the traffic and circumstances of a testing period

Solution

(1) Make sure all testing experiences get the same traffic mix. (2) Avoid special commerce events for A/B testing (Christmas,

Black Friday, Valentine’s day etc.). (3) In case you have a seasonal business, A/B test your hypotheses

in both on and off season.

3@parezem | #MeasureCamp

Cross-device A/B Testing

Challenge

Attribution in cross-device A/B testing. One user, different devices, not the same testing experience

Solution

(1) Use targeting only to one device type - not solve multiple same device type issue

(2) Wait when tools add “user-centric testing”

@parezem | #MeasureCamp

Long Purchase Decision Making Process

Challenge

Customers from your A/B test made the actual decision before your A/B test was launched

Solution

(1) Target new visitors only. (2) Set micro-conversion goals when an A/B test focuses on an

early part of the purchase process

First visit on the LP

T = 0 T + 28

Purchase!Third visit on the LP

T + 12

Your A/B is launched

T + 20

Research

Finances, laziness, more research, discount coupons

@parezem | #MeasureCamp

Research Online, Purchase Offline

Challenge

Your online A/B tests influence offline purchases too

Solution

(1) Show discount coupons for offline purchases - each testing experience has unique discount coupon. Experience A = DSC08AExperience B = DSC08B

@parezem | #MeasureCamp

Optimising for Maximum CLTV

Challenge

You don’t want to get more average customers. You do want to get more excellent customers!

Solution

(1) A survey after a purchase - “Would you recommend us to your friend?” (2) Re-evaluate your A/B tests after few months. (3) Patterns with your historical data - predict CLTV. (4) Immediate insights: e.g. a share of auto-renewal customers

buy again recommend

stay loyal upgrade

@parezem | #MeasureCamp