Lean Startup Experiment Canvas

Lean Experiment CanvasA Framework for Continuous Process

Improvement & Experimentation

Chad Cote T-2

*Adapted from the Business Model Canvas by Alexander Osterwalder (https://strategyzer.com/) &

the Lean Canvas by Ash Maurya (https://leanstack.com/blog/)

CreationDate: UpdateDate: Owner:ExperimentTitle:

Lean Experiment Canvas

AssumptionsPredictionFalsifiableHypothesis

ResultsofExperiment

Risks

KeyStakeholdersKey Metrics

ExperimentalSteps

NextObstacles

Timeframe

TheQuestion

Background

CreatedbyChad CoteLeanExperimentCanvasisadaptedfromTheBusinessModelCanvas(www.strategyzer.com)undertheCreativeCommonsAttribution-Share

“Important issues should be presented in writing. Nothing so sharpens the thought process as writing down one’s arguments. Weaknesses overlooked in oral discussion rapidly become painfully obvious on the written page.” - Admiral Rickover

Think Like a Scientist, Start-up Founder, and Engineer

Ideas (are cheap)

Execution (is everything)

How do I get from hereto there?

Ideas (are cheap)



Validated Learning

Validated Learning“the process of demonstrating

empirically that a team has discovered valuable truths

about a startup’s present and future business prospects” -

Eric Ries

Ideas (are cheap)



Validated Learning

CreationDate:

UpdateDate:

Owner:

ExperimentTitle:


Assumptions

Prediction

FalsifiableHypothesis

ResultsofExperiment

Risks

KeyStakeholders

Key Metrics

ExperimentalStepsNextObstacles

Timeframe

TheQuestion

Background


Ideas (are cheap)


Operations / Management / Perseverance / Luck / Timing / etc..

Validated Learning

CreationDate:

UpdateDate:

Owner:

ExperimentTitle:


Assumptions

Prediction

FalsifiableHypothesis

ResultsofExperiment

Risks

KeyStakeholders

Key Metrics

ExperimentalStepsNextObstacles

Timeframe

TheQuestion

Background


Focus: Understanding Reality

“Toyota Kata” - Mike Rother

“Lean Startup” - Eric Ries

“Certain to Win” - Chet Richards

OODA Loop

PDSA & BML loops are simple manifestations of a

much larger thought process contained in the OODA

“loop”

Experiments are the “Genes” of Innovation & Continuous Process Improvement

PDSAs are a process of probing and assessing your environment to uncover

better methods/ideas

Experiment Loop = Learning

Most Basic Building Block of Improvement Good Experiments lead to learning which is passed down to the organization over

time

Minimizebeing wrong * time

Maximal Interestingess

#1 #2

…. #n

Experiment Loop = Learning

Time

Drive Experimentation in the Direction of Maximal Interestingness

OOHyTOOHyT

OOHyT

Directed Opportunism

Observe - Orient - Hypothesis - Test: OOHyT


#1 #2

….

#nTime

….

Speed != Agility

Goal: Maintain Accurate Representation of Reality

The “faster” you learn the more accurate your orientation

AgilityOOHyT

OOHyT

OOHyT

OOHyT

Directed Opportunism


The direction of maximal interestingness can be very rapidly updated to reflect new information, by evolving the rough consensus. …. It is enough to figure out experimental next steps. This ability to reorient and adopt new mental models quickly (what military strategists call a fast transient) is at the heart of agility. - “Breaking Smart” http://breakingsmart.com/en/season-1/

Maximize Agility Through Experimentation Loops

Plan

Do

Study

Act “Natural” pathway to fill out the canvas



Assumptions(leapsoffaith)Prediction (expectation)FalsifiableHypothesis(If{Idothis},then{this} willhappen)

ResultsofExperiment (validatedlearning)

Risks

KeyStakeholders(connections)Key Metrics(process, outcomes)

Experimental Steps

NextObstacles

Timeframe(prepare, test)

TheQuestion

Background(goandsee)

2

3

7

4 6

8

11 12

1

95

10

Plan

Do

Study

Act DoWhere is the “Do” stage?

“Boots on the ground”Get out of the office!



Assumptions(leapsoffaith)Prediction (expectation)FalsifiableHypothesis(If{Idothis},then{this} willhappen)

ResultsofExperiment (validatedlearning)

Risks

KeyStakeholders(connections)Key Metrics(process, outcomes)

Experimental Steps

NextObstacles

Timeframe(prepare, test)

TheQuestion

Background(goandsee)

2

3

7

4 6

8

11 12

1

95

10

“No plan survives contact with the enemy.” - Helmuth von Moltke

“The thinking–doing loop is kept as short as possible so as to reduce uncertainty and increase tempo.” - The Art of Action

Example Used Throughout the Presentation Call Center Wait Times




ResultsofExperiment

Risks


ExperimentalSteps

NextObstacles

Timeframe

TheQuestion

Background

Call Center Wait Times 3/7/16 4/22/16 Chad Cote & Call Center Supervisor

After creating and implementing a call center dashboard we realized we had substantial wait times occurring in the early am hours. Wait times were up to 14 minutes on Monday mornings and we had high average wait times on a daily basis.

How can we reduce average wait times in the early am hours?

-By moving labor hours from the pm to the am we can shorten wait times w/o large risk to pm wait times

-We have enough staff to handle the current volume

-We are betting on the relationship between utilization and cycle time to hold

-We have staff who are ok with changing their work schedules

-We have a good call center script that we believe works to handle call volume

If we bring in one staff member earlier in the morning and allow them to leave earlier in the afternoon

then we can reduce average wait time in the am hours which will also drive down wait times on average.

-Drive down wait times in the early am

-Wait times in pm hours won't be negatively affected

-Afternoon wait times could rise substantially creating supply and demand mismatches

-Staff actually don't want to change working hours and this may hurt morale

-We could be short on total staff and moving staff hours around will only shift the constraint

-Call center operators

-Department Leadership

-Patients who call the call center

-Downstream departments who will be the "receivers" of new appointments made by patients in the am

1. Talk to staff members who would be most likely to come in the early am hours

2. Change hours unofficially for two weeks to test the change

3. Run the test for two weeks and analyze the results on a weekly basis

4. If successful, officially change the staffing hours through the appropriate means

Average wait times in the am hours

Total average wait times

Call center wait time complaints

Test: 2 weeks

-Huge success driving down wait times in the early am hours from 14 minutes to 5 minutes-Successful in driving down overall total wait times (see graph)-Staff actually prefer working the am hours -Wait times in the afternoon hours were not negatively affected confirming the assumption of the relationship between CT and utilization-Extended the test phase a few weeks to ensure process stability

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

2015

-36

2015

-37

2015

-38

2015

-39

2015

-40

2015

-41

2015

-42

2015

-43

2015

-44

2015

-45

2015

-46

2015

-47

2015

-48

2015

-49

2015

-50

2015

-51

2015

-52

2015

-53

2016

-2 20

16-3

2016

-4 20

16-5

2016

-6 20

16-7

2016

-8 20

16-9

2016

-10

2016

-11

2016

-12

2016

-13

2016-14

2016

-15

2016

-16

2016

-17

AverageWaitTimeinCallCenter

AvgMinutesonHold Series2 Series3 Series4

-Morning am times are still the constraint even though we made huge gains

-We found out that staff spend some time performing their annual trainings during this time frame and that should be eliminated. This might be a "Just Stop It" next step

-Look for new bottlenecks, such as lunch and be aware that with more employees arriving at the same time lunch hours will have to be managed appropriately

Go & SeeSeeing the right problem and defining it

accurately

Know your customers

Understand the greater context of what you are trying to accomplish

What is the “as-is” condition? What is the impetus to change? Why would anyone care?




ResultsofExperiment

Risks


ExperimentalSteps

NextObstacles

Timeframe

TheQuestion

Background

After creating and implementing a call center dashboard we realized we had substantial wait times occurring in the early am hours. Wait times were up to 14 minutes on Monday mornings and we had high average wait times on a daily basis.

“One good question can give rise to several layers of answers,

can inspire decades-long searches for solutions, can

generate whole new fields of inquiry, and can prompt changes in entrenched thinking. Answers,

on the other hand, often end the process.” - Stuart Firestein

How can we reduce average wait times in the early am hours?

?? ? ? ?…

hypo

thes

ishy

poth

esis

hypo

thes

is

hypo

thes

ishy

poth

esis

hypo

thes

is

hypo

thes

ishy

poth

esis

hypo

thes

is

hypo

thes

ishy

poth

esis

hypo

thes

is




ResultsofExperiment

Risks


ExperimentalSteps

NextObstacles

Timeframe

TheQuestion

Background

Hypo

thesis1

Hypo

thesis2

Hypo

thesis3

Hypo

thesis4

Hypo

thesis5

Hypo

thesis6

Hypo

thesis7

Hypo

thesis8

Hypo

thesis9

Hypo

thesis10

Hypo

thesis11

Hypo

thesis12

Hypo

thesis13

Uncertainty

PrioritizeUncertainyReductionPrioritize Uncertainty Reduction

If {I do this}, then {this will happen}

Must be able to be proven false (i.e. not astrology)

If we bring in one staff member earlier in the morning and allow them to leave earlier in the afternoon

then we can reduce average wait time in the am hours which will also drive down wait times on average.

What you think will happen

Looking for cause and effect relationships

Perform experiments that explicitly test your “stories”

Never say, “Let’s Just See What Happens”

-Drive down wait times in the early am

-Wait times in pm hours won't be negatively affected

Expose your beliefs and assumptions about the underlying

problem

What do you believe internally that makes you believe this hypothesis and experiment is the right one?

Your assumptions inform and shape your hypothesis

-By moving labor hours from the pm to the am we can shorten wait times w/o large risk to pm wait times

-We have enough staff to handle the current volume

-We are betting on the relationship between utilization and cycle time to hold

-We have staff who are ok with changing their work schedules

-We have a good call center script that we believe works to handle call volume

Your experiment is more than likely biased and based on beliefs and assumptions

that are wrong

There is potential for negative results. Write them

down.

-Afternoon wait times could rise substantially creating supply and demand mismatches

-Staff actually don't want to change working hours and this may hurt morale

-We could be short on total staff and moving staff hours around will only shift the constraint

Process Metrics&

Outcome MetricsTight feedback loops

Action/Information need to be tightly coupled

Clear cause & effect

**Call centers have very clear metrics which makes testing easier

Average wait times in the am hours

Total average wait times

Call center wait time complaints

Understand who will be influenced by

experimental changes

Beware of externalities

-Call center operators

-Department Leadership

-Patients who call the call center

-Downstream departments who will be the "receivers" of new appointments made by patients in the am

Be explicit about the steps of the

experiment

1. Talk to staff members who would be most likely to come in the early am hours

2. Change hours unofficially for two weeks to test the change

3. Run the test for two weeks and analyze the results on a weekly basis

4. If successful, officially change the staffing hours through the appropriate means

How long will it take to prepare?

How long will it take to run the experiment?Test: 2 weeks

“An experiment that successfully proves a

hypothesis is a measurement; one that doesn’t is a

discovery. A discovery, an uncovering— of new

ignorance.” - Enrico Fermi

-Huge success driving down wait times in the early am hours from 14 minutes to 5 minutes-Successful in driving down overall total wait times (see graph)-Staff actually prefer working the am hours -Wait times in the afternoon hours were not negatively affected confirming the assumption of the relationship between CT and utilization-Extended the test phase a few weeks to ensure process stability

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

2015

-36

2015

-37

2015

-38

2015

-39

2015

-40

2015

-41

2015

-42

2015

-43

2015

-44

2015

-45

2015

-46

2015

-47

2015

-48

2015

-49

2015

-50

2015

-51

2015

-52

2015

-53

2016

-2 20

16-3

2016

-4 20

16-5

2016

-6 20

16-7

2016

-8 20

16-9

2016

-10

2016

-11

2016

-12

2016

-13

2016-14

2016

-15

2016

-16

2016

-17

AverageWaitTimeinCallCenter

AvgMinutesonHold Series2 Series3 Series4

Looking for confirmatory data,

and more importantly contradictory data

Risks

• Observe• Orient• Decide• Act

-Morning am times are still the constraint even though we made huge gains

-We found out that staff spend some time performing their annual trainings during this time frame and that should be eliminated. This might be a "Just Stop It" next step

-Look for new bottlenecks, such as lunch and be aware that with more employees arriving at the same time lunch hours will have to be managed appropriately

Start Experimenting

for a copy of the Lean Experiment Canvas email me

at:leanmeanimprovingmachine@

gmail.com