OSDC 2015: Bernd Ahlers | What is your configuration management system doing?

Bernd Ahlers – Graylog, Inc. [email protected]

What is your configuration management doing?

Bernd AhlersGraylog, Inc.


● Bernd Ahlers● Working at Graylog, Inc. in Hamburg/Germany● [email protected]● twitter.com/berndahlers● github.com/bernd

About me


Configuration management tools allow us to manage our computing resources in an

effective and consistent way

Introduction


They make it easy to run hundreds or thousands of machines without having to

manually execute the same tasks over and over again

Introduction


By using shared modules/cookbooks it is pretty easy to end up with hundreds of

managed resources like files, packages and services per node

Introduction


Nodes can be configured to check for updates and to apply new changes

automatically

Introduction


This helps us to roll out changes to lots of nodes very easily but also makes it

possible to quickly break our infrastructure resulting in outages

Introduction


So being able to collect, analyze, and monitor all events that happen sounds like

a good thing to me

Introduction


It makes it possible to quickly analyze if an incident has been caused by changes from

a config management system

Introduction


Having the errors thrown during an update run available in a central place can help

debugging problems

Introduction


Graphing the duration of config management runs can help finding unusual

behavior

Introduction


Do you know what your configuration management is doing?

Introduction


Example implementation withChef, Icinga2, Graylog

Introduction


Tools: Chef

www.chef.io


Popular open source configuration management tool

Uses Ruby to provide a powerful DSL for defining and managing resources

Tools: Chef


Tools: Icinga2

www.icinga.org


Free and open source monitoring toolUse it, it is good

Tools: Icinga2


Tools: Graylog

www.graylog.org


Free and open source tool for centralized log management and analysis

Tools: Graylog


● Started by Lennart Koopmann in his free time in 2010 (Graylog2 at that time)

● TORCH GmbH founded as company behind Graylog in late 2012

● Big rewrite that got released as 0.20 in Feb 2014

Tools: Graylog


● New US based company Graylog, Inc. founded in Jan 2015

● Renamed from Graylog2 to Graylog● Graylog 1.0 release in Feb 2015

Tools: Graylog


Getting events from Chef into Graylog




Need to build a consumer for the internal event bus

More than 60 event callbacks available



run_start run_started run_completed run_failed ohai_completed skipping_registration registration_start registration_completed registration_failed node_load_start node_load_failed run_list_expand_failed node_load_completed cookbook_resolution_start cookbook_resolution_failed cookbook_resolution_complete cookbook_clean_start removed_cookbook_file cookbook_clean_complete cookbook_sync_start synchronized_cookbook updated_cookbook_file cookbook_sync_failed cookbook_sync_complete library_load_start library_file_loaded library_file_load_failed library_load_complete lwrp_load_start lwrp_file_loaded lwrp_file_load_failed lwrp_load_complete attribute_load_start attribute_file_loaded attribute_file_load_failed attribute_load_complete definition_load_start definition_file_loaded definition_file_load_failed definition_load_complete recipe_load_start recipe_file_loaded recipe_file_load_failed recipe_not_found recipe_load_complete converge_start converge_complete resource_action_start resource_failed_retriable resource_failed resource_skipped resource_completed resource_current_state_loaded resource_current_state_load_bypassed resource_bypassed resource_up_to_date resource_update_applied resource_updated stream_opened stream_closed stream_output handlers_start handler_executed handlers_completed provider_requirement_failed whyrun_assumption msg



Output show during a Chef run is also based on that



[2015-04-21T13:51:01-07:00] INFO: Forking chef instance to converge...[2015-04-21T13:51:01-07:00] INFO: *** Chef 12.2.1 ***[2015-04-21T13:51:01-07:00] INFO: Chef-client pid: 1789[2015-04-21T13:51:07-07:00] INFO: Setting the run_list to ["recipe[osdc-2015-demo::default]", "recipe[osdc-2015-demo::chef]", "recipe[osdc-2015-demo::webserver]"] from CLI options[2015-04-21T13:51:07-07:00] INFO: Run List is [recipe[osdc-2015-demo::default], recipe[osdc-2015-demo::chef], recipe[osdc-2015-demo::webserver]][2015-04-21T13:51:07-07:00] INFO: Run List expands to [osdc-2015-demo::default, osdc-2015-demo::chef, osdc-2015-demo::webserver][2015-04-21T13:51:07-07:00] INFO: Starting Chef Run for ubuntu1404[2015-04-21T22:52:06+02:00] INFO: template[/etc/rsyslog.conf] sending restart action to service[rsyslog] (delayed)[2015-04-21T22:52:06+02:00] INFO: service[rsyslog] restarted



Consuming Chef events requires a Start Handler that listens for events



Implement all the callbacks?



Too noisy – for now onlySuccessFailure

Resource modification



Sending Chef events as GELF messages over HTTP



GELFJSON based format to send structured

events to a Graylog server



Event consumer has access to Chef runtime data and adds that to the events



{ "_chef_version": "12.2.1", "_chef_run_id": "d4b07325-c431-4c98-be2c-a292afda1b30", "_chef_event_name": "run_completed", "_chef_node_name": "ubuntu1404", "_chef_node_environment": "_default", "_chef_node_roles": "", "_chef_node_recipes": "webapp::default", "_chef_run_start_time": "2015-04-21T21:22:45.075Z", "_chef_run_end_time": "2015-04-21T21:22:45.500Z", "_chef_run_elapsed_time": 0.424473734, "_chef_run_total_resources": 6, "_chef_run_updated_resources": 0, "_chef_run_cookbooks": "webapp", "version": "1.1", "timestamp": 1429651365.5025952, "host": "ubuntu1404", "short_message": "Chef run succeeded on node ubuntu1404"}



{ "_chef_version": "12.2.1", "_chef_run_id": "74a2c04b-66f1-47d3-8c36-aaf04d9d9030", "_chef_event_name": "run_failed", "_chef_node_name": "ubuntu1404", "_chef_node_environment": "_default", "_chef_node_roles": "", "_chef_node_recipes": "webapp::default", "_chef_run_start_time": "2015-04-21T21:24:39.969Z", "_chef_run_end_time": "2015-04-21T21:24:41.679Z", "_chef_run_elapsed_time": 1.710085302, "_chef_run_total_resources": 6, "_chef_run_updated_resources": 1, "_chef_run_cookbooks": "webapp", "_chef_error_message": "service[apache2] (webapp::default line 5) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'", "_chef_error_type": "Mixlib::ShellOut::ShellCommandFailed", "_chef_error_backtrace": "/opt/chef/embedded/lib/ruby/gem ...", "version": "1.1", "timestamp": 1429651481.684527, "host": "ubuntu1404", "short_message": "Chef run failed on node ubuntu1404"}



{ "_chef_version": "12.2.1", "_chef_run_id": "a0b68697-9286-4193-9569-1a1939a24c5f", "_chef_event_name": "resource_updated", "_chef_node_name": "ubuntu1404", "_chef_node_environment": "_default", "_chef_node_roles": "", "_chef_node_recipes": "webapp::default", "_chef_resource_action": "create", "_chef_resource_identity": "/etc/apache2/sites-enabled/webapp.conf", "_chef_resource_name": "template", "_chef_resource_cookbook_name": "webapp", "_chef_resource_cookbook_version": "0.1.0", "_chef_resource_recipe_name": "default", "_chef_resource_declared_type": "template", "_chef_resource_defined_at": "webapp::default line 18", "_chef_resource_duration": 0, "version": "1.1", "timestamp": 1429651713.1174529, "host": "ubuntu1404", "short_message": "Resource updated: template"}



Using GELF HTTP input in Graylog to receive events via HTTP



Using HTTP makes it easy to implement the same for other configuration

management systems



Install gem and configure Start Handler in chef-client configuration file

https://rubygems.org/gems/chef-handler-graylog

/etc/chef/client.rb



begin require 'chef/handler/graylog/gelf_start_handler'

# IP address and port of the GELF HTTP input on # your Graylog server. graylog_server_url = 'http://10.0.2.2:12201/gelf' options = {}

start_handlers << Chef::Handler::Graylog::GelfStartHandler.new( graylog_server_url, options)rescue Object => e Chef::Log.error( "Loading Graylog start handler failed: #{e.message}")end



All Chef events will now be sent to Graylog as GELF messages over HTTP


DEMO

DEMO

https://github.com/Graylog2/osdc-2015-demo


Finish

Source code for the event handler and the demo setup available on GitHub

https://rubygems.org/gems/chef-handler-grayloghttps://github.com/Graylog2/chef-handler-graylog

https://github.com/Graylog2/osdc-2015-demo

Contributions welcome!


Thank you!

Thank you for your attention!


QA

QA

Ask me anything!

[email protected]@berndahlers

Technology

OSDC 2015: Bernd Ahlers | What is your configuration management system doing?