Cloud Tech III: Actionable Metrics

Actionable MetricsEnabling Decision-Making in Netflix’s Decentralized

Environment

Cloud Tech IIIOctober 6, 2012Roy Rapoport

@royrapoport, [email protected]

Thursday, October 18, 12

mailto:[email protected]

mailto:[email protected]

Me

• Been in tech for about 20 years

• Systems engineering, networking, software development, QA, release management

• Time at Netflix: 1195 days (3y:3m:1w)

• (Current) job at Netflix: Make things better (Security Monkey, Python Platform, Central Alert Gateway, Breaking Stuff.. )


Metrics Humor


Metrics Humor


Metrics Humor


Metrics Humor


Metrics Humor

% of instances with even public IP addresses


Technology Overview


Technology Overview• SoA, REST, Mostly Java



• Simple overall architecture:








Culture Overview


Culture Overview

• Freedom and Responsibility


Culture Overview


• Distributed Operations


Culture Overview


• Distributed Operations

• Get out of the way of Developers


The Metric Lifecycle



•Send



•Send

•Look



•Send

•Look

•Alert


Systems

• Flexible

• Scalable

• Self-Service


TelemetryFlexible, Scalable, Self-Service

import netflix.metrics[...] self.nm = netflix.metrics.Metrics("core_cag")[...]def api(self): self.nm.nfCounter("api") [...] self.nm.nfCounter(“application_%s” % application)[...]


VisualizationFlexible, Scalable, Self-Service












AlertingFlexible, Scalable, Self-Service



• Static vs Dynamic Thresholds



• Static vs Dynamic Thresholds

• Compare to history


For Example ...

What the ...

Last 3 hours’ core_tools.core_cag_api


For Example ...Visualization (Continued)

Last 4 days’ core_tools.core_cag_api

even more questions!



Last 10 days’ core_tools.core_cag_api

What caused the spike?



Show alert volume per application

Someone had a rough few days...


Don’t Like Surprises...{ "alerts": [ { "applyTo": "cluster", "condition": { "minPercent": 90.0, "noise" : .2, "maxPercent": 25.0, "type": "DoubleExponential" }, "metricName": "core_cag_api", "severity": "major" } ], "clusters": [ "core_tools" ]}


Threshold Tuning

• An Abbreviated History ...


Threshold Tuning(in the beginning)

Some priests offer their prayers to alien creatures best left forgotten. This ill-advised worship twists their minds in odd ways. Overlords find these warped men useful due to the unnatural powers they can channel. The dark priests most favored by their strange gods have powerful protections, and defeating one of them is sure to bring down a terrible curse upon the victor.

- http://www.descentinthedark.com/_d_/dark_priests.php


http://www.descentinthedark.com/_d_/dark_priests.php



• Systems owned by IT








• Want an alert? Submit a ticket








• Want an alert? Submit a ticket

• Want to tune an alert? Submit a ticket






Threshold Tuning(It gets better)



• You get to configure your own threshold




• Freedom!




• Freedom!

• Also, you have to configure your own thresholds


Threshold Tuning(Are we there yet?)



• Play with historical data




• Huge difference




• Huge difference

• Still falls short


Threshold Tuning(Yeah, that’s the ticket)



• Computers can be good at this















If Time Allows ...


Events vs Metrics


Events vs Metrics

• Irregular Interval


Events vs Metrics


• Point in time


Events vs Metrics


• Point in time

• Lack magnitude


Why Build It?


Why Build It?

• Change management

• Vs Change control


Why Build It?



• What Changed?


Why Build It?



• What Changed?

• Better Alerting


Chronos


Chronos

• Rapidly Prototyped


Chronos


• Adapters and reporters


Chronos



• Easy querying


Chronos



• Easy querying

• Alarming

• Something happened


Chronos



• Easy querying

• Alarming


• ... X times in Y minutes


Chronos



• Easy querying

• Alarming


• ... X times in Y minutes

• Something didn’t happen


Chronos



• Easy querying

• Alarming

• Medium volume


Chronos



• Easy querying

• Alarming

• Medium volume

• Recursive

• Recursive


End Result


End Result

• Massive decrease in change control tickets


End Result


• Not talking about SOX or PCI


End Result



• Better visibility into changes


End Result




• Decreased TTR


End Result




• Decreased TTR

• Especially for bad code deployments


End Result




• Decreased TTR

• Especially for bad code deployments

• You should do this


I Didn’t Mention

• End-to-end testing and alerting

• External availability and performance

• Open Connect

• Jobs


Questions?
