Icinga Camp San Diego 2016 - Unrealized Role of Monitoring

The UnrealizedRole of:

Monitoring& Alerting

@jasonhand | VictorOps | #DevOpsDays

JasonHandDevOps Evangelist

VictorOps@jasonhand | VictorOps | #DevOpsDays

SCaLE 14xSouthern

California

Linux

Expo



2015MonitoringSurvey@jasonhand | VictorOps | #DevOpsDays


Why Are You Collecting This Data?NOTE: You may choose more than one

» Performance analysis and trending

» Fault and Anomaly detection

» Capacity Planning

» A/B Testing

» We don’t do anything with collected metrics


The ResultsNOTE: Respondents may have chose more than one

» Performance analysis and trending - 63%

» Fault and Anomaly detection - 53%

» Capacity Planning - 45%

» A/B Testing - 11%

» We don’t do anything with collected metrics - 3%


Tyranny of the

S.L.A.(Service Level Agreement)@jasonhand | VictorOps | #DevOpsDays

HighAvailabilityPrediction & Prevention@jasonhand | VictorOps | #DevOpsDays



That's Important

... but ...@jasonhand | VictorOps | #DevOpsDays



Business

Objectives?


Happy Camper


Customerswant more than just

99.999% Uptime@jasonhand | VictorOps | #DevOpsDays


Where's the

Innovation?@jasonhand | VictorOps | #DevOpsDays

? =@jasonhand | VictorOps | #DevOpsDays

? =ContinuousImprovement


How Important is

Learning &Innovation?@jasonhand | VictorOps | #DevOpsDays





The result of underutilizing monitoring & alertingis that the IT department and the organization haveno chance to...

learn,improve, orinnovate.@jasonhand | VictorOps | #DevOpsDays

Continually understanding & responding to the feedbackfrom

monitoring, logging, & alerting

allows you to use information about events in the past to drive future actions.


SwitchingGears




It's not just about

Prediction& Prevention


Respond &Repair...Quickly@jasonhand | VictorOps | #DevOpsDays

Nope


MTTRRather Than

MTBF@jasonhand | VictorOps | #DevOpsDays

Failure IsInevitable


us·er/ˈyoozər/Distributed fault injection test suite for production.

credit: Leon Fayer (@papa_fire)@jasonhand | VictorOps | #DevOpsDays

Successis a result of

Failure@jasonhand | VictorOps | #DevOpsDays

Understand

LearnInnovate


re·sil·ient/rəˈzilyənt/The ability to resist, absorb, recover from or successfully adapt to adversity or a change in conditions


Changecan cause failure

but innovation requires

Change


Conflict


ChangeRequired


“Without deviation from the norm, progress is not possible ”Frank Zappa


What Did You

LearnFrom the Recovery Efforts?(including monitoring & alerting)


Postmortems / Learning Reviews:Stories of:

What took placeleading up to & duringthe disruption & recovery efforts


Who was

involved?


What did they

see?@jasonhand | VictorOps | #DevOpsDays

What was

said?@jasonhand | VictorOps | #DevOpsDays

What

actionswere taken?jhand.co/chatopsbook


How do events & actions

correlateover time?@jasonhand | VictorOps | #DevOpsDays

5 Why's@jasonhand | VictorOps | #DevOpsDays

What is the "cause"of the Problem?

Root Cause is ...


Our...

obsession with

"Root Cause"@jasonhand | VictorOps | #DevOpsDays

Asking "why".. leads to ..

Blame


Blamingleads to..

operators hiding relevant & important information


We must

believethat our operators are doing their best given theconstraints of the "system"


"We are here to"

LearnFrom Failure(and success)


Rather than ..


AvoidFailure


What's the

Story?@jasonhand | VictorOps | #DevOpsDays

InnovateLearning from both success & failureto develop & implementsmall incremental improvementsis critical.


LearningOrganization


Learning does NOT come from

Reading&Listening@jasonhand | VictorOps | #DevOpsDays

Learning comes from

Doing@jasonhand | VictorOps | #DevOpsDays

Real Learning comes from:

ObservingOrientingDecidingActingJohn Boyd's OODA Loop


Example:

Learning to play the

Dobro Guitar@jasonhand | VictorOps | #DevOpsDays


Learning


Why?Go from knowing...to understanding...to learning

NOTE:(Requires making mistakes)



“We will trade some uptime in exchange for innovation-Dave Hahn (Netflix)”

DevOpsDays Boise 2016(today)


Are We Doing

itRight?@jasonhand | VictorOps | #DevOpsDays

What do your

Postmortemslook like?

Are they setting you up to learn?


"The Story"-Timeline-Who Was Involved-Context

(Seeing, Saying, Executing)

-Action Items

(Small Incremental Improvements)


Shift our gazefrom:

maintaining& protecting


LearningWhich leads to...

Improving& Innovating


we increase value

of monitoring & alertingof the IT teamsof Products & Services& of the Organization.


HypothesizeExploreStretchExperimentFailLearnTry Again@jasonhand | VictorOps | #DevOpsDays


Learning & Innovatingleads to uncovering new ways of

building, deploying, and maintaining software & infrastructureWhich leads to...


ResilientSystems


The

By-product

of a highly

resilientsystem is ...



HighlyAvailablesystem@jasonhand | VictorOps | #DevOpsDays

The UnrealizedRole of:

Monitoring& Alerting is ....


Learning&

Innovation


Thank

YouBe Victorious!


References:

Monitoring Survey: https://kartar.net/2015/08/monitoring-survey-2015---metrics/Firefighter: https://www.learyfirefighters.org/wp-content/uploads/2013/09/cover-slide-1.jpgMechanic: https://upload.wikimedia.org/wikipedia/commons/4/4b/Flickr_-_Israel_Defense_Forces_-_Airplane_Technician,_March_2010.jpgGnome Plan: http://www.nerdfitness.com/wp-content/uploads/2012/04/Screen-Shot-2012-03-30-at-3.15.38-AM-1024x7591.jpgNOC: https://upload.wikimedia.org/wikipedia/commons/


References:

Kodak: http://file.answcdn.com/answ-cld/image/upload/v1/tk/brand_image/b59911fc/91d6e71d30a0878dfe3cb30a22751cb874a3ea8c.jpegVW Camper: https://upload.wikimedia.org/wikipedia/commons/d/d7/VW_Camper.jpgBlockbuster: https://jordanandeddie.files.wordpress.com/2013/11/blockbuster-feature.jpgBorders: http://smashingtops.com/wp-content/uploads/2012/06/borders_logo1.jpg


References:

Chained Hands: https://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&ved=0ahUKEwjgrNCDh5TMAhXJs4MKHaoZDssQjBwIBA&url=http%3A%2F%2Fwww.publicdomainpictures.net%2Fdownload-picture.php%3Fadresar%3D50000%26soubor%3Dhands-in-chains.jpg%26id%3D40426&bvm=bv.119745492,d.amc&psig=AFQjCNFIdnDPzSqiLA-znIW5SCTCUHhqEw&ust=1460926880336203Inevitable: http://vignette4.wikia.nocookie.net/matrix/images/5/51/SMITH.png/revision/latest?cb=20110214092002


References:

Accident Free:http://www.compliancesigns.com/media/digital-scoreboard/1000/Safety-Awareness-Sign-DSE-195271000.gifStewie:http://chroniclesofredmark.com/wp-content/uploads/2014/01/Stewie.gifchange: http://i.imgur.com/EQyC6N3.gifHard drive: https://i.imgur.com/pWsKSEf.gifChange: https://farm6.staticflickr.com/5208/5270199049df99b234e9od.jpgValue: https://d13yacurqjgara.cloudfront.net/users/



Technology

Icinga Camp San Diego 2016 - Unrealized Role of Monitoring