Lessons Learned From A/B Testing Wins and Losses

WHO I AM: GROWTH HACKER

ANDY

Growth Team/groTH tem/ n.

A group within the company that develops the product experience in a way that creates more activated users

! Pre Product/Market fit: build the growth engine

! Post Product/Market fit: optimize the engine (make it bigger)

! Any time: generate hacks, non-obvious ways to reach your goals

Definition

Building the Testing Engine

1

GET THE PIECES IN PLACE BEFORE ADDING GAS

Do things that don’t scale

Test yourself to product market fit

What type of product are our users interested in?

How can we reach this users the best way?

Pour Gas on it

WE HAD TO EXPERIMENT WITH A LOT OF THINGS

How do you build what users actually want?

Child ID

Dog TrackerFamily Locator

Lost Phone

Ads Landing Pages

THINK OF IT AS DIVIDE & CONQUER

Optimizing the Test Engine

2

RISKLOW HIGH

Big Swings

Optimizations

REW

ARD

HIGH

Start Ups need step function take bold bets

Scalable growth through testing

Lather, Rinse, Repeat. Continuous Improvement

Evaluate Results

Test Design

Develop Hypothesis

Implement Leanings

Developing hypotheses

How to Develop Hypotheses1. Leverage the team, institutional knowledge,

understand important “Themes”

2. Qualitative studies, surveys

3. Quantitative analysis

Test Design

Test Design

How to Design Tests Effectively1. Determine right testing format

2. Optimize test to avoid overinvesting

3. Execute quickly, full instrumentation for analysis

Evaluate Results

How to Evaluate Results1. Analyze core metrics impact of each variant

2. Ensure all numbers are statistically significant

3. Understand long term impacts and side effects

Evaluate Results

Implement Learnings

How to Implement Learnings1. Understand what caused the change

2. Determine how this learnings applies to product aspects

3. Promote visibility of learnings/takeawaysImplement Leanings

Qualitative Tests Painted Door Tests Feature Tests

INVESTMENT COSTLOW HIGH

Painted Door

Top of Funnel Test

Full FeatureMany Some Few

Types of tests avoid overinvesting

Qualitative Tests

PurposeDetermine interest in product via conversations with your users and potential users

KeyTest as many ideas as possible but keep in mind that people want to please you

ExampleEmergency ContactsETA Sharing Carpooling

Qualitative Tests

Feature Tests

PurposeBuilding a functional feature and see how users interact

KeyDon’t build the full feature, start at the top of the funnel and work your way down

ExampleEmergency ContactsRecommenderFUE

Add EC

Send EC

Accept EC

Create ACT

(Partial) Feature Tests

Aggressive vs. Passive Tests

Passive Tests End Point Aggressive Tests

LEARNINGSLOW HIGH

HOW TO TRUST YOUR RESULTS?

Test Design - Explicit Control Variant Allocation

Design variants to have an additional control group

➔ Ensures control group meets test user criteria➔ Reduce risk of variant assignment issues

Test Design - Explicit Control Variant Allocation

All Users

Test Design - Explicit Control Variant Allocation

Eligible for experimentIneligible for experiment

Test Design - Explicit Control Variant Allocation

Eligible but excluded from experiment

Ineligible for experiment Included in experiment

Test Design - Explicit Control Variant Allocation

Treatment 1

Eligible but excluded from experiment

Ineligible for experiment

Control Variant 1

Variant 2

Test Design - Explicit Control Variant Allocation

Always consider keeping long-term holdout groups

➔ Understand potential impacts that may not be seen early in the user lifecycle

➔ Enables measurement of how impacts rollup into product growth

“A GOOD PLAN, VIOLENTLY EXECUTED TODAY, IS BETTER THAN A PERFECT PLAN EXECUTED NEXT WEEK.”

GENERAL GEORGE S PATTON

HERE IS WHAT I AM UP TO NEXT

Andreas@reddit.com