Experimentation-Driven Product Development

Robin Pam

Sr. Product Marketing Manager Optimizely

Product Manager Optimizely

Jamie Connolly

With Optimizely X Full Stack

Robin Pam

Sr. Product Marketing Manager Optimizely

Product Manager Optimizely

Jamie Connolly

Agenda

● Why Experimentation Matters ● What Is Experimentation-Driven

Product Development? ● How to Build an Experimentation

Platform ● Full Stack Demo

• We are recording today’s presentation

• You will receive a copy of the slides after the webinar

• Please submit questions via the text box

Housekeeping

Innovation in the physical world can take decades

3 YEARS LATER

Innovation by the hour

At any given point in time, there isn't just one version of Facebook running, there are probably 10,000.

Mark Zuckerberg““

A B

A

B

9,800 Google search experiments in 2016

www.google.com/search/howsearchworks/

1,653 Google search launches in 2016

www.google.com/search/howsearchworks/

I haven’t failed. I’ve just found 10,000 ways that won’t work.” Thomas Edison

“

Experimentation-Driven Product Development

medium.com/netflix-techblog

uber.com/blog/colorado

LAUNCHBUILDDES IGN

PRAY

EXPER IMENTITERATE

LAUNCHBUILD

DES IGN PRAY

EXPER IMENTITERATE

LAUNCHBUILDDES IGN PRAY

EXPER IMENT ITERATE

LAUNCHBUILDDES IGNPRAY

EXPER IMENTITERATE

LAUNCHBUILDDES IGN PRAY

EXPER IMENTITERATE

DES IGN BUILD EXPER IMENT ITERATE

LAUNCHPRAY

How to build an experimentation platform that is fast and scales

// ab testing is easyif random() < 0.5: var = ‘a’else: var = ‘b’

PlanOutSixPack

Proctor

phpA/B

ConvertAirlock

Genetify

FeatureBee

AbbaAlephBet

abclub

Confidence

FluidFeatures

Gertrude

Rollout

Split

TrebuchetVanity

Wasabi

from planout.experiment import SimpleExperiment from planout.ops.random import * class FirstExperiment(SimpleExperiment): def assign(self, params, userid): params.button_color = UniformChoice(choices=['#ff0000', '#00ff00'], unit=userid) params.button_text = WeightedChoice( choices=['Join now!', 'Sign up.'], weights=[0.3, 0.7], unit=userid) my_exp = FirstExperiment(userid=12) # parameters may be accessed via the . operatorprint my_exp.get('button_text'), my_exp.get('button_color')

medium.com/netflix-techblog

medium.com/netflix-techblog

Analytics

medium.com/netflix-techblog

medium.com/netflix-techblog

eng.uber.com/data-viz-intel/

Traffic Splitting Remote Configuration Analytics

Traffic Sampling Audience Targeting

Mutual Exclusion State Persistence

Multivariate Treatment Global Holdout

QA Tool Automation Whitelisting

Logging

Permissions Feature Toggles Feature Rollouts

Kill Switch Scheduling

Staging Environments Audience Definitions

Webhooks REST API Audit Trail

Event Dispatching Event Storage Sessionization

Statistical Methods Metric Definitions

User Aliasing Segmentation DW Integration Notifications Visualizations

3 key ingredients for robust experimentation

1. Trust“How can I trust the experiment was set up and tracked correctly?”

2. Process“What are the best instrumentation practices to minimize technical debt?”

3. Maintenance“The engineer who built this left the company… now what?”

Three key challenges with experimentation

Demo

Full Stack Use Cases

Feature Flags Performance Optimization

Algorithms

User Experience Roll outs Pricing

$$

Marketing Channels

How many experiments do you run per year?

How many changes do you launch per year?

Your Product

Roadmap is your

Experiment Roadmap

Join our next webinar: Getting Started with Server-Side Testing

December 13, 2017

https://optimize.ly/serversidewebinar

Q&ARobin Pam Jamie Connolly