Upload
puppet
View
102
Download
0
Embed Size (px)
Citation preview
The Five Dirty Words of CI
J. Paul Reed Release Engineering Approaches PuppetConf, 2017
“CI”
@jpaulreed #PuppetConf
“CI”
@jpaulreed #PuppetConf
@jpaulreed #PuppetConf
J. Paul Reed@jpaulreed on
Alum of The Ship Show
15+ Years in Build/Release Engineering
Now, a DevOps™ Consultant™
Master of Science candidate in Human Factors & System Safety
@jpaulreed #PuppetConf
Root Cause Analysis
Dirty Word #1
@jpaulreed #PuppetConf
@jpaulreed #PuppetConf
Root Cause AnalysisA method of problem solving used for
identifying the root causes of faults or problems.
A factor is considered a root cause if removal thereof from the problem-fault-sequence prevents the final undesirable
event from recurring.— Wikipedia
@jpaulreed #PuppetConf
Our Perception
@jpaulreed #PuppetConf
Our Reality@jpaulreed #PuppetConf
“We found the Root Cause!”
@jpaulreed #PuppetConf
“Root” “Cause”Cause is something
you construct. What you call “root cause”
is simply the place where you stop looking any further.
— Sidney Dekker@jpaulreed #PuppetConf
A Better Choice: “Root Cause Analysis”
Proximate Cause(s)
But…@jpaulreed #PuppetConf
The “Five Whys”
Dirty Word #2
@jpaulreed #PuppetConf
The “Five Whys”Five Whys is an iterative interrogative
technique used to explore the cause-and-effect relationships
underlying a particular problem. The primary goal of the technique is to determine the root cause of a defect or
problem by repeating the question "Why?" — Wikipedia@jpaulreed #PuppetConf
What “Five Whys” Always Feels Like to Me@jpaulreed #PuppetConf
The Perception: Incidents are deterministic, like code: same inputs, same outputs.
Every time.@jpaulreed #PuppetConf
A Crash Where?!
int foo(object& r) { r.Blah(); return 1; }
@jpaulreed #PuppetConf
The Operational Reality@jpaulreed #PuppetConf
A Better Choice: “Five Whys”
Just… no. “Swiss Cheese” model
Systemic model
@jpaulreed #PuppetConf
Human Error
Dirty Word #3
@jpaulreed #PuppetConf
Human ErrorHuman error has been cited as a primary cause or contributing factor in disasters and accidents
in industries as diverse as nuclear power, aviation, space exploration, and medicine.
Prevention of human error is generally seen as a major contributor to reliability and safety of
(complex) systems.— Wikipedia@jpaulreed #PuppetConf
But Really: What Is Human Error?
— James Reason’s conception@jpaulreed #PuppetConf
Who gets to draw “the line?” What incentives/interests do they have in putting that “line” where it is? It ignores other stories or even the possibility of entertaining other explanations…
Isues with “Human” “Error”
@jpaulreed #PuppetConf
“Human” “Error”Human error is not the cause of
failure, but the effect.
So, human error… can never be the conclusion of your investigation.
It is the starting point.— Sidney Dekker@jpaulreed #PuppetConf
Human Error often a prelude to a constraint on learning: “Well, just fire the dumb, bad apples… problem solved!”
@jpaulreed #PuppetConf
Were the World So Simple…
@jpaulreed #PuppetConf
A Tiny Problem in “The Cloud”
@jpaulreed #PuppetConf
A Different Take on Failure
@jpaulreed #PuppetConf
A Huge Opportunity to LearnOther operational tools with no input sanity checks The Service Health Dashboard’s real dependencies Indexing Subsystem’s insufficient partitioning Indexing Subsystem hadn’t been fully restarted for years
Had Amazon “just fired” this engineer, they would have never learned these critical details
about their system or how to operate it@jpaulreed #PuppetConf
A Better Choice: Human Error
Stop Saying It. Then, Keep
Not-Saying It.@jpaulreed #PuppetConf
“Why didn’t you?”/ “You should
have…”
Dirty Word #4
@jpaulreed #PuppetConf
Counterfactuals
Dirty Word #4
@jpaulreed #PuppetConf
CounterfactualsCounterfactual thinking is a concept in psychology that
involves the human tendency to create possible alternatives to life events that have already occurred.…
Counterfactual thinking is, as it states, "counter to the facts." These thoughts consist of the "What if?" and the "If I had only..." that occur when thinking of how
things could have turned out differently.— Wikipedia@jpaulreed #PuppetConf
A Waste of Time
@jpaulreed #PuppetConf
Discussing a reality that does not exist.
@jpaulreed #PuppetConf
A Better Choice: Counterfactuals
Don’t.
@jpaulreed #PuppetConf
Best Practice
Dirty Word #5
@jpaulreed #PuppetConf
@jpaulreed #PuppetConf
Best PracticeA best practice is a method or technique
that has been generally accepted as superior to any alternatives because it produces results that are superior to
those achieved by other means or because it has become a standard way of
doing things.— Wikipedia@jpaulreed #PuppetConf
Only the Best of Practices
@jpaulreed #PuppetConf
Only the Best, Artisanal of Practices“Best” is superlative Best practices in complex systems often ignore context Best practices are often not completely defined (especially in complex systems)
@jpaulreed #PuppetConf
Only the Best, Artisanal of Practices“Best practice,” by definition, leaves no space for innovation / discovery Relatedly (and maybe worst of all) it discourages experimentation
@jpaulreed #PuppetConf
Best Practicin’ Ourselves Outta Business
@jpaulreed #PuppetConf
Only the Best, Artisanal of Practices“Best practice,” by definition, leaves no space for innovation / discovery Relatedly (and maybe worst of all) it discourages experimentation Best practice applies to a domain little of our work exists in
@jpaulreed #PuppetConf
Obvious
ComplicatedComplex
Chaotic
Disorder
@jpaulreed #PuppetConf
Obvious
ComplicatedComplex
Chaotic
Governing Constraints
Good Practice
Rigid Constraints Best Practice
Lack of Constraints Novel Practice
Enabling Constraints Emergent Practice
Disorder
@jpaulreed #PuppetConf
Obvious
ComplicatedComplex
Chaotic
Governing Constraints
Good Practice
Rigid Constraints Best Practice
Lack of Constraints Novel Practice
Enabling Constraints Emergent Practice
Disorder
@jpaulreed #PuppetConf
@jpaulreed #PuppetConf
A Better Choice: Best Practice
“Good Practice” Or ensure you apply
“best practice” in the correct domain
@jpaulreed #PuppetConf
“Yeeahhh... so what kind of… Continuous Improvement...
would you... say ya do here?”@jpaulreed #PuppetConf
The Path of Continuous Improvement
is not “linear” (Nor is it “one-and-done”)
Takeaway I
@jpaulreed #PuppetConf
Respect Reality
Takeaway II
@jpaulreed #PuppetConf
Treat People Like the Professionals
They Are
Takeaway III
@jpaulreed #PuppetConf
Go Forth and Continuously Improve
J. Paul Reed www.jpaulreed.com
@jpaulreed
www.release-approaches.comSimply Ship. Every Time.@jpaulreed #PuppetConf
@jpaulreed #PuppetConf
Broken Build
Dirty Word #1
@jpaulreed #PuppetConf
“Flappers”
Dirty Word #2
@jpaulreed #PuppetConf
bobs-mac-mini. local
Dirty Word #3
@jpaulreed #PuppetConf
Merge Window
Dirty Word #4
@jpaulreed #PuppetConf
Jenkins Build Number
Dirty Word #5
@jpaulreed #PuppetConf