Embed Size (px)
DESCRIPTION
At Rackspace, we're implementing DevOps principles to enable scale and faster flow of business value, for us that means customer satisfaction. There's lots of work for us to do to complete our implementation, but this talk shows a little about our belief system for DevOps and how we're trying to change monitoring into a source of customer delight.
Citation preview
1
MONITORING AT SERVICE PROVIDER SCALE June 23, 2014
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
INTRODUCTIONS
CHRIS JACKSON Rackspace CTO (DevOps Practice)
@chriswiggy
+44 (0) 7712 854462
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 3
Image courtesy of http://www.toastmasters.krakow.pl/
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4
THE DEVOPS RACK APPLIANCE…
ALL OF THE THINGS IN 3-CLICKS!
Image courtesy of Axiomtek
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 5
HOW DO WE APPLY DEVOPS TO SCALE SERVICE?
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
THE DEVOPS PRACTICE AT RACKSPACE
6
THE FIRST WAY – CREATE A DELIVERY SYSTEM
THE SECOND WAY – AMPLIFY FEEDBACK LOOPS
THE THIRD WAY – LEARN & EXPERIMENT CONTINUOUSLY
This is PRODUCT & SUPPORT delivering great service powered by automation
!
Our DevOps practice is a way to accelerate feedback by removing complexity
!
The goal is to never hold our customers up, we must always be faster…
!
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
SCALING TRANSFORMATION
7 SCALING TRANSFORMATIONS
Image courtesy of Wordpress
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 8
SO WHAT ABOUT MONITORING?
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 9
WHY #MONITORINGSUCKS FOR RACKSPACE
We generate over 40,000 events per second on just our Cloud Monitoring platform.
If a customer is Managed, we troubleshoot ANY alarm that is raised.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 10
WHY #MONITORINGSUCKS FOR RACKSPACE
Image courtesy of Spot On Lists
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 11
WHY #MONITORINGSUCKS FOR RACKSPACE
Image courtesy of Cepolina
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 12
WHY #MONITORINGSUCKS FOR RACKSPACE
Image courtesy of New Line Cinema
RACKSPACE® HOSTING | WWW.RACKSPACE.COM 13
WHAT ARE WE DOING TO FIX THIS?
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
SCALE MONITORING TUNING
14
WHEN TUNING IS MANUAL: EFFORT=tN
Where t = time to configure a single node and N = number of nodes
0 100 200 300 400 500
Effo
rt
# Nodes
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
SCALE MONITORING TUNING
15
WHEN TUNING IS INTELLIGENT: EFFORT=t1-N
Where t = time to configure a single node and N = number of nodes Ef
fort
# Nodes
Shrink this!
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
TOOLS FOR INTELLIGENT CONFIGURATION
16
Configuration Management If you know what you want to measure!If your constraint is just the number of nodes!Gets complicated when you’re thresholding all of the things…!
Configuration Discovery We call it Waldo, you know it as Sartori in Openstack!Uses ohai-solo to find information about your configuration!Compare to best practice and make changes accordingly…!
Image courtesy of Walker Books Ltd
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
MONITORING DATA ANALYTICS
17
EVENT GENERATION EVENT PROCESSING
EVENT HANDLING
EVENT ANALYTICS
TUNING INSIGHT
CREATE FEEDBACK LOOPS INSIDE MONITORING REMOVE WORK BY TUNING ALARMS
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
INTELLIGENT ALERTS
18
ANOMALY MONITORING PATTERN RECOGNITION
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
WHAT ARE WE USING?
19
statsd
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
TOOLCHAIN INTEGRATION
20
EVENT GENERATION EVENT PROCESSING
EVENT HANDLING
EVENT ANALYTICS
TUNING INSIGHT
statsd
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
HELP US TO DO THE HARD WORK
21
YOU
US
Things We Want to Know™!How do you use monitoring?!
How intelligent is the tuning?!
What use of anomaly or threshold?!
Approach for creating tool chains?!
What do you dislike doing?!
We’ll build it and open source it!!
Image courtesy of Dave Hogan/Getty Images
Twitter - @chriswiggy http://www.rackspace.com/devops http://devops.rackspace.com
22
THANK YOU
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
MORE THAN A CLOUD COMPANY
WE PROVIDE
SUPPORT FOR EVERY STEP OF YOUR JOURNEY
JUST STARTING? RACKSPACE ADVISORY
SERVICES
NEED SERVICES? RACKSPACE HYBRID
CLOUD
NEED SUPPORT? RACKSPACE DEVOPS
AUTOMATION SERVICE
