Embed Size (px)
Citation preview
Are we compliant?Auditing Change Management Policies
with Splunk and Puppet
http://bit.ly/puppetsplunkslides
Jeff [email protected]
1Monday, October 11, 2010
Jeff McCune
• Joined Puppet Labs in May, 2010
• Former SA at Netsmart Technologies
• Solaris / RedHat Web App Infrastructure
• Human Health Information Systems
• HIPPA, SAS 70 Type II Compliance
2Monday, October 11, 2010
What’s this all about?
• Audits are a fact of life
• Systems drift
• Puppet Master manifests change
• The logs provide no link
• Puppet and Git in synchrony with Splunk
3Monday, October 11, 2010
Fun with Regulations
• Increased focus on compliance
• SAS 70
• HIPPA
• IPA
• PCI DSS
• etc, etc...
4Monday, October 11, 2010
Compliance is Easy
Golden VM
Clones
5Monday, October 11, 2010
Drifting in and out of Compliance
Follow procedures
Justify the change
Firefighting
Inevitable
Constant drift
6Monday, October 11, 2010
The Trouble with Time
• Are we compliant?
• right now?
• last week?
• last year?
• Why weren’t we?
• Why is this difficult?
7Monday, October 11, 2010
Advanced Management
• We have next-generation tools
• Puppet
• Git
• Subversion
• Splunk
• Redmine
8Monday, October 11, 2010
Two major issues
• Propagation
• Time
9Monday, October 11, 2010
Larry’s commit
Change PropagationMany
Nodes
a872b46
10Monday, October 11, 2010
Time
“Why did that one thing happen that one time?”
11Monday, October 11, 2010
Bridge the GapEvents Commits
12Monday, October 11, 2010
The Missing Link
• puppetmasterd -‐-‐config-‐version \ /demo/get-‐config-‐version-‐script
• [root@puppet ~]# /demo/get-‐config-‐version ref="refs/heads/jeff" commit="b585f7fe"
• Jeff ’s processor, --reports=logversionShould ship with puppet “soon”
13Monday, October 11, 2010
get-config-version#! /bin/bashset -‐uset -‐ecd /demo/puppet-‐demotoolsref="$(git symbolic-‐ref HEAD)"if [[ -‐f .git/"${ref}" ]]; then commit="$(cat .git/${ref})"else commit="UNKNOWN"fiecho "ref=\"${ref}\" commit=\"${commit}\""
14Monday, October 11, 2010
logversion.rb
# Create logversion.rb by copying log.rbdef process self.logs.each do |log| saved_message = "#{log.message}" log.message << " " << log.version Puppet::Util::Log.newmessage(log) log.message = saved_message
endend
15Monday, October 11, 2010
Untagged Events
16Monday, October 11, 2010
Tagged Events
17Monday, October 11, 2010
Who to blame?
Blame this guy The commit proves it
18Monday, October 11, 2010
Putting it all together
• Demo time!
19Monday, October 11, 2010
Steps to Reproduce
• Fork and clone puppet-‐demotools on github
• logversion.rb goes into/usr/lib/ruby/site_ruby/1.8/puppet/reports
• --config_version /path/to/your/script
• --reports=logversion,store
• Make sure syslog catches daemon.* and splunk is indexing syslog
• Note: syslog outputs are off with -‐-‐verbose
20Monday, October 11, 2010
Future Work• Commit hooks into puppetmasterd
activation and the ticketing system
• Splunk URLs to redmine, trac, salesforce...
• Closed loop from business case to system modification by puppet.
21Monday, October 11, 2010
Questions?
• Google Moderator
• http://bit.ly/arewecompliant?
• http://bit.ly/puppetsplunkslides
• Twitter: 0xEFF
• Email: [email protected]
22Monday, October 11, 2010
