Mean Time to Sleep: Quantifying the On-Call Experience

@lozzd • @ryan_frantz

Mean Time to SleepQuantifying the on-call experience

Laurie Denness@lozzd

Ryan Frantz@ryan_frantz


Who is in an on-call rotation?


Who is on call right now?


Who feels like on-call sucks?

Welcome. How is on call?


Let’s help our people sleep


Make on-call more bearable


Incremental Changes


Email toAcknowledge


Email to Acknowledge• Replying “ack” with some context makes it appear in

IRC too



IRC too



IRC too



IRC too



IRC too



IRC too



IRC too


Email Only Alerts• Do you care if RAID becomes degraded in the middle of

the night?



the night?

• Do you care if one of your web/hadoop/X boxes dies in the middle of the night?



the night?

• Do you care if one of your web/hadoop/X boxes dies in the middle of the night?

• Can it wait until the morning?


Added Context• Previous service state

• Duration in that state


• Previous service state




• Previous service state









• Alert recipients













• Notes





• Notes

• Link to Runbook





• Notes

• Link to Runbook





• Notes

• Link to runbook


Alert Storms• Reduce noise when 200 things go wrong by aggregating


Alert Storms• Reduce noise when 200 things go wrong by aggregating

• Trigger alert percentage of pool over threshold


Low friction downtime• IRC commands to downtime hosts/sets of hosts


Low friction downtime• IRC commands to downtime hosts/sets of hosts


Downtime Reminders• Help prevent false notifications


Downtime Reminders• Help prevent false notifications


Event Handlers• Teach Nagios to augment the team



• Restarting services (nscd)




• Re-running jobs (transient errors)




• Re-running jobs (transient errors)

• Duplicate crons (Chef)


Incremental Improvements?• Maybe



• More ideas; hoped they’d stick



• More ideas; hoped they’d stick

• We didn’t know because we didn’t measure


Measure Everything• “You can’t manage what you can’t measure.”

- Deming (not really)


Measure Everything• “You can’t manage what you can’t measure.”

- Deming (not really)

• But, we weren’t measuring anything


What should we measure?


What should we measure?• Volume of alerts (total, by severity)



• Alert categorization (actionable vs not)




• Alert times: Off-hours?




• Alert times: Off-hours?

• Noisy hosts/services


Opsweekly

@lozzd • @ryan_frantz We have data.


Aggregate alerts1. Look at reports



2. Wow, look at all those alerts for the same thing




3. Aggregate alerts




3. Aggregate alerts

4.Profit


Parent relationships• Prevent alerts due to upstream issues (downed switch)



• Standard Nagios feature



• Standard Nagios feature

• Computers can do this for us!


Parent relationships• signalvnoise.com



• LLDP on host shows switch info




• Put switch info into Chef using ohai




• Put switch info into Chef using ohai

• Create Nagios host configs based on data


Service Dependencies• Hundreds of Graphite-sourced checks


Service Dependencies• Hundreds of Graphite-sourced checks

• Created new template that sets a servicegroup that depends on the Graphite service.


Keep on analyzing• It’s okay to just identify and delete alerts that don’t

mean anything!


Keep on analyzing• It’s okay to just identify and delete alerts that don’t

mean anything!

• Or move them to email only


More Quantification!


Reviewing the Year• Use reports



• Use search



• Use search

• Identify noisiest alerts


Reviewing the YearYEARLY REPORT SCREENSHOTS


• Great time to look at this data and make improvements

Nagios Hack Day/Week


• Great time to look at this data and make improvements

• If Disk Space is the worst. Can we rethink that?

Nagios Hack Day/Week


Outsource Your Alerts• Etsy’s Search Team has on-call rotation



• A whole subset of alerts that don’t go to Ops



• A whole subset of alerts that don’t go to Ops

• More teams starting this but Search Team is at 100%


Sleep Tracking


“Track your life!” - @ph






Did it work?


Did it work?


Did it work?• Yes.





• Signal to noise ratio is much better





• Okay, so it’s a little more complicated than that




• Adding alerts all the time means new “annoying” things




• Adding alerts all the time means new “annoying” things

• Keep monitoring


What’s next?


• We focus on people’s sleep

The Effect of Sleep



• But not the effect on the person when they come to work the next day

The Effect of Sleep




• How do we measure the impact of sleep loss/deprivation?

The Effect of Sleep




• How do we measure the impact of sleep loss/deprivation?

The Effect of Sleep

• Subjective: Pittsburgh Sleepiness Scale

• Objective: Psychomotor vigilance task (PVT) to measure alertness


Beyond Opsweekly• Employee wellness program


Beyond Opsweekly• Employee wellness program

• Security have started using past sleep data to check for weird logins to systems


More context: nagios-herald


More reports• We have a bunch of data, we can build better reports,

drill down to analyze alerting trends


More reports• We have a bunch of data, we can build better reports,

drill down to analyze alerting trends

• Can we attribute particular actions to reduced noise volume?

• Aggregate alerts

• Non-downtimed alerts


Thanks


Etsy Ops Team


SewMona


Open Source/Links• http://ryanfrantz.com/mtts

• https://github.com/etsy/opsweekly

• https://github.com/etsy/nagios-herald

• https://github.com/jonlives/jawboneup_to_graphite

• http://codeascraft.com


Questions?





Engineering

Mean Time to Sleep: Quantifying the On-Call Experience