Embed Size (px)
Citation preview
4/3/15
@evan2645
EVAN GILMAN
Bloated Chefs A Tale of Gluttony, and the Path to Enlightenment
4/3/15BLOATED CHEFS
4/3/15
Agenda
BLOATED CHEFS
1. Chef resources in use at PD
2. Problems encountered as we grew
3. Measuring chef-client run
4. How we fixed it
5. How fast is it now?
4/3/15BLOATED CHEFS
CHEF @ PAGERDUTY
4/3/15BLOATED CHEFS
PD CHEF RESOURCES
4/3/15
pd_iptables
BLOATED CHEFS
4/3/15
pd-ipsec::policies
BLOATED CHEFS
4/3/15
sumo_source
BLOATED CHEFS
4/3/15
pd_datadog_alert
BLOATED CHEFS
4/3/15BLOATED CHEFS
4/3/15BLOATED CHEFS
ALL WAS NOT WELL
4/3/15
As we grew…
BLOATED CHEFS
4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs
4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs
• Awkward pauses at the beginning of the run
4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs
• Awkward pauses at the beginning of the run
• chef-client run took several minutes
4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs
• Awkward pauses at the beginning of the run
• chef-client run took several minutes
• chef-client OOM
4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs
• Awkward pauses at the beginning of the run
• chef-client run took several minutes
• chef-client OOM
4/3/15BLOATED CHEFS
4/3/15BLOATED CHEFS
4/3/15BLOATED CHEFS
MEASURING
4/3/15
Measuring Run Time
BLOATED CHEFS
4/3/15
Measuring Run Time
BLOATED CHEFS
https://github.com/joemiller/chef-handler-profiler
4/3/15
Measuring Resources
BLOATED CHEFS
• Total number of resources per run, by type
• Number of updated resources per run, by type
4/3/15
Measuring Memory
BLOATED CHEFS
• Gather proc stats with sys-proctable
• Gather GC stats
• Can be emitted as statsd
4/3/15
Measuring Memory
BLOATED CHEFS
4/3/15BLOATED CHEFS
WHAT WE FOUND AND
WHAT WE DID
4/3/15
Step-through Searches
BLOATED CHEFS
4/3/15
Step-through Searches
BLOATED CHEFS
From this
4/3/15
Step-through Searches
BLOATED CHEFS
From this
To this
4/3/15
Step-through Searches
BLOATED CHEFS
417MB -> 190MB
4/3/15
Step-through Searches
BLOATED CHEFS
417MB -> 190MB~54%
4/3/15
Partial Searches
BLOATED CHEFS
4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled
4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled
• hash2node
4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled
• hash2node
• Two searches touched
4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled
• hash2node
• Two searches touched
90s -> 60s
4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled
• hash2node
• Two searches touched
90s -> 60s30%
4/3/15
Result Memoization
BLOATED CHEFS
4/3/15
Result Memoization
BLOATED CHEFS
• Common search data
4/3/15
Result Memoization
BLOATED CHEFS
• Common search data
• API-backed LWRP’s
4/3/15
Result Memoization
BLOATED CHEFS
• Common search data
• API-backed LWRP’s
• Can be generalized
4/3/15
Result Memoization
BLOATED CHEFS
• Common search data
• API-backed LWRP’s
• Can be generalized
4/3/15
API Tarpitting
BLOATED CHEFS
4/3/15
API Tarpitting
BLOATED CHEFS
Centralize calls
4/3/15BLOATED CHEFS
OTHER NASTIES
4/3/15
Other Nasties
BLOATED CHEFS
• Too many conditional guards
4/3/15
Other Nasties
BLOATED CHEFS
• Too many conditional guards
• tmpfs storage
4/3/15
Other Nasties
BLOATED CHEFS
• Too many conditional guards
• tmpfs storage
• Multiple package resources (Chef 12)
4/3/15
Other Nasties
BLOATED CHEFS
• Too many conditional guards
• tmpfs storage
• Multiple package resources (Chef 12)
Six seconds for twelve packages
4/3/15BLOATED CHEFS
BEFORE/AFTER
4/3/15
Memory Saved
BLOATED CHEFS
Before:
After:
4/3/15
Memory Saved
BLOATED CHEFS
Before: ~500MB
After:
4/3/15
Memory Saved
BLOATED CHEFS
Before: ~500MB
After: ~60MB
4/3/15
Memory Saved
BLOATED CHEFS
Before: ~500MB
After: ~60MB
88% less memory!
4/3/15
Seconds Saved
BLOATED CHEFS
Before:
After:
4/3/15
Seconds Saved
BLOATED CHEFS
Before: ~180s/run
After:
4/3/15
Seconds Saved
BLOATED CHEFS
Before: ~180s/run
After: ~30s/run
4/3/15
Seconds Saved
BLOATED CHEFS
Before: ~180s/run
After: ~30s/run
~84% faster!
4/3/15BLOATED CHEFS
FREEDOM
4/3/15
Thank you.
@evan2645
EVAN GILMAN
