Experimenting in the Enterprise Agile 2013 Conference

Experimenting

Enterprise

in the

David J BlandStartups consulted: 18Twitter followers: 2179Iterations of Coaching: 215

Speaker Vanity Metrics

Brian BozzutoBlogs & Articles Written: 77Presentations Delivered: 46Teams coached: 54

This session doesn’t guarantee a 100 million dollar success, but it should help prevent a 100 million dollar failure.

A business model used to be static and survived many years.

Internet

Facebook

0 5 10 15 20 25 30 35 40

Years to Reach a Market of 50 Million

The Lean Startup provides a scientific approach to creating and managing startups and get a desired product to customers’ hands faster.

-Eric Ries

Entrepreneur Intrapreneur

Skills

This is great in theory, but it’s not for B2B.

This is great in theory, but I’m a visionary.

EntrepreneurIntrapreneur

Define the value proposition for:

Source: Adapted from Steve Blank

Decider

B2C B2B

Are your users/customers:

Source: Adapted from Steve Blank

Aware of problem?Actively seeking a solution?Hacking together a solution?Have $$$ for your solution?

Can We Build It?

Should We Build It?

Requirements = Hypotheses

Simple“Everyone Knows That”

Complicated“Let’s consult the expert”

Complex“Whoa, it all makes sense now…”

Chaos“That still doesn’t make sense”

Work with experts to inform our hypothesis

Run experiments to gain new knowledge

Safe Fail != Fail Safe

Experiment withproduct & process

If you experiment just to see what happens, then you’ll succeed at seeing what happens.

Be careful how you structure your experiments

Then here

Start hereLast here

NowExecuteQuickly

Experiment Template

For <customer segment> I predict <outcome>

when I run <experiment>

Experiment Example

For new customers, I predict 10% increase in

conversion when I offer a “no questions asked” cancellation policy

Experiment Guidelines

• Customer Segment (who)• Up to / Limit Of (how many)• Length of Time (how long)• Financial Exposure (how much)• Amplification & Dampening Measures

Experiment DashboardID Hypothesis Description Status Next Steps

We predict that adding

a refund feature will

increase conversions

Story 13c adds refund label and call to

action button that puts

customer in contact with call center

agent.

Complete

0.2% increase in conversion on cohort. Meeting

with VP of Ops to discuss

strategy.

“This is all great, but how do we know what

experiments to run?”

Iterative Product+ Static Business Model

Failed Organization

? ? ???

“Let’s map out our existing business model”

Source: Adapted from Alex Osterwalder

tip: tell a story

? ? ???

“Let’s validate our riskiest assumptions”

This is most likely your Riskiest Assumption

This is most likely your other Riskiest Assumption

What happens when we subtract costs from revenue?

“Okay, now we have a big list of assumptions, how do we decide where to start?”

experiment

“Let’s dot vote!”(you could, but that’s pretty subjective)

experimentexperiment

experiment

(ok better, but what are they tied to?)

Now we have a focusing mechanism

focustime time

experiments experiments

And assumptions that need validation

assumption assum

assumption assumption

time time

This is Experiment Mapping

assumption assum

time time

Customer Segment is our focus

assumption assum

time time

CustomerSegment

Will they pay for our product?

assumption assum

time time

CustomerSegment

They willpay for our

product

Can we scale to meet demand?

assumption assum

time time

CustomerSegment

product

We can scaleto meetdemand

Can we use our distribution channels?

assumption assum

time time

CustomerSegment

product

We can useexisting

distributionchannels

Are we solving a real need?

assumption assum

time time

CustomerSegment

product

Our productsolves a need

they have

We can useexisting

Which of these is the riskiest?

assumption assum

time time

CustomerSegment

product

they have

We can useexisting

Can we do a concierge test?

assumption assum

time time

CustomerSegment

product

they have

We can useexisting

Conciergetest with 10customers

We could build a prototype

assumption assum

time time

CustomerSegment

product

they have

We can useexisting

Build aprototype

Maybe run a focus group?

assumption assum

time time

CustomerSegment

product

they have

We can useexisting

Build aprototype

Run a focusgroup with80 people

Prototype will take time

assumption assum

time time

CustomerSegment

product

they have

We can useexisting

Build aprototype

Focus groups take a long time too

assumption assum

time time

CustomerSegment

product

they have

We can useexisting

Build aprototype

Concierge test is quick & we’ll learn

assumption assum

time time

CustomerSegment

product

they have

We can useexisting

Build aprototype

“Let’s swarm on this experiment”

highlearning

shorttimeframe

Just because you can measure it, does not mean you should measure it.

Auditable

Actionable

Accessible

Source: Eric Ries The Lean Startup

Auditable

“Be able to defend your metrics with confidence”

Auditable?

zombieproject

metrics

Actionable

“Does this metric incentivize a change in behavior?”

Actionable?

Facebook likes won’t influence your dog

Actionable

Number of steps should influence

your lifestyle

Actionable?

What does this tell you about your

product?

Accessible

“Can you easily access and share this data?”

Accessible?

Not accessible

Accessible

Image Source: Lean.st

AcquisitionActivationRetentionReferralRevenue

“Metrics are people, too”

Source: Eric Ries The Lean Startup

Hopeful

Satisfied

PassionateSource: Adapted from Cooper & Vlaskovits

Hopeful

SatisfiedPassionate

Acquisition

Activation

RetentionReferral

Start small.

When you have stories around an uncertain feature, add a hypothesis that states the expected outcome on a specific metric.

If you have a Kanban Board, add a Validate Column to

the very right of it.

Problem Solution Process

Known Known Waterfall orAgile

Known Unknown Agile

Unknown Unknown Lean Startup

Where are the unknown / unknown’s in your enterprise?

David J BlandExperimentor

Brian BozzutoPrincipal

Thank You!

Experimenting in the Enterprise Agile 2013 Conference

Technology

AGILE: THE AGILE GUIDE TO THE ENTERPRISE

Best Practices for Enterprise Agile Transformation Paper_BP for Enterprise Agile...Best Practices for Enterprise Agile Transformation. ... provides organizations with the essential

SI_INCOSE_2013_The Agile Enterprise

THE AGILE ENTERPRISE - PMI KC Mid-America Chapter · AGILE ENTERPRISE: AGILE TOP-TO-BOTTOM • Agile Strategy • Agile Portfolio, Program, Project Management • Agile Execution

Lean Transformation: Thinking, Learning, & Experimenting For … · 2010-05-03 · lean.org Lean Enterprise Institute 1 Lean Transformation: Thinking, Learning, & Experimenting For

Tampere Goes Agile - Experimenting Through Change Keynote

Enterprise Scaled Agile - IBM · PDF fileEnterprise Scaled Agile Overview Overview of Enterprise Scaled Agile methodology, the SAFe ... The Scaled Agile Framework

Experimenting with agile to maximize value delivery

Agile for the Enterprise

7 Misconceptions of Enterprise Agile - Modern Analystmedia.modernanalyst.com/7 Misconceptions of Enterprise Agile.pdf · Training for Enterprise Agile 33 Course Duration Agile Fundamentals

Exploring Enterprise Agile Transformation Strategies Mike Cottmeyer, Enterprise Agile Coach LeadingAgile, LLC

enterprise agile lean modeling

Agile Enterprise Data Warehousingceit.aut.ac.ir/~90131914/BI/Lecture Slides/Agile Enterprise Data... · Agile Enterprise Data Warehousing ... From Chaos to Architecture Data Mart

Agile Enterprise Partnerships

Scaling agile. Agile across the enterprise

Agile enterprise integration

AGILE for Enterprise Series #1* · AGILE for Enterprise Series #1* 2 Agile in the Enterprise #1 Nitin Khanna, PMI-ACP, ICP-ACC ... The Agile Alliance, 2001 Principles behind the Agile

Agile in Enterprise - mini Italian Agile Day

Enterprise Agile Adoption

Introduction to Enterprise Agile Frameworksprojectmanager.org/images/downloads/PDC_2014_Presentations/... · Which Enterprise Agile Frameworks are having traction? •Scaled Agile