Managing ExternallyGenerated Files with Puppet
Kevin PaulisseDevOps Engineer and Team Lead
Collective, Inc.
Hello, I am Kevin Paulisse
Background:
• Linux guy for 17+ years, Puppet devotee for 3+ years• Current position: DevOps engineer and team lead @ Collective• Prior: Shopbop (Amazon), Philips, State of WI, TDS, Alliant Energy
Contact information:
• Work e-mail: [email protected]• Personal e-mail: [email protected]• Twitter: @kpaulisse• Github: kpaulisse• LinkedIn: https://www.linkedin.com/in/kpaulisse
[email protected]@gmail.com@kpaulissehttps://github.com/kpaulissehttps://linkedin.com/in/kpaulisse
About Collective
What Collective does:
• Serve Ads: 10's of thousands of requests/sec• Bid on Ads: 100's of thousands of requests/sec• Data Science: 1000's of predictive models weekly
Collective's technology stack:
• Linux (physical & virtual)• Puppet and Foreman• 3 new data centers this year• 300 node Hadoop cluster
Collective is hiring:
• DevOps/Linux Engineers• Data Scientists, Devs, etc.
[email protected]://www.collective.com
Agenda
• Introduction
• Kerberos crash course
• Managing Kerberos keytab files with Puppet:
• Traditional approaches: file{} and exec{}
• Other approaches to generating keytabs
• Our implementation
• Scaling the solution to multiple Puppet masters
• Other use cases
Kerberos crash course: Terminology
Kerberos: Networkauthentication protocol
Principal: An account
Keytab: A file containingencrypted keys forprincipals
Host Keytab: On Linux,/etc/krb5.keytab
Kerberos crash course: Keytab files
Each host needs its own Kerberos principal and corresponding keytab.
A Kerberos Administrator creates and manipulates principals.
Task NeedsAdmin
ScramblesPassword
Determine if there is a Kerberos principal for the host
Create the Kerberos principal if none exists
Save Kerberos credentials to a keytab file
(Scrambling password refers to MIT Kerberos -- other systems may differ.)
Installing Kerberos keytab with file{}
file { "/etc/krb5.keytab": ensure => present, owner => "root", group => "root", mode => 0400, source => WhatGoesHere?}
● The keytab file must already exist somewhere
● The above resource does not generate the file
Installing Kerberos keytab with exec{}
exec { "create-etc-krb5.keytab": command => "kadmin -w S3cr3tPass ...", creates => "/etc/krb5.keytab", user => "root",}
● The exec resource runs on the agent
● Each agent gets to see your administrative credential
Generate keytab asynchronously: Design
Keytab generation
● Use the admin credential on just one server
● Generate keytab before Puppet runs (put keytab where Puppet can find it)
● Store the keytabs in such a way that a host can see only its own keytab
Puppet manifest
● Use a file{} resource to define content/source
● Host keytab in catalog (not admin credentials)
● Usual file{} features: set permissions, restart services, etc.
Generate keytab asynchronously: Implementation
Manual processDocs: "Don't forget to run the keytab script"
Management system (e.g. The Foreman)Generate keytab via a "hook" from management system when a host is added to the configurationor a Puppet run is scheduled
Task: Generate a keytab file before the Puppet run
Generate keytab during Puppet run: Design
Keytab generation
● Keytab has to be generated during the Puppet run
● Keytab has to be available on Puppet master so it can be included in the catalog
● Don't use exec{} on agent to generate keytab
Puppet agent
● Use a file{} resource to define content/source
● Host keytab in catalog (not admin credentials)
● Usual file{} features: set permissions, restart services, etc.
Internals of a Puppet run
Read Further: https://docs.puppetlabs.com/puppet/3.7/reference/subsystem_agent_master_comm.html
1. SSL certificate generation and signing
2. Plugin sync
3. Build and fetch catalog (pictured at left)
a. HTTPS POST with facts
b. Catalog = everything but file sources
4. Apply (and fetch files where needed)
5. Send reportPOST /catalog
Manifests
Hiera
Compiler
Templates
Send factsReceive catalog
Apply catalog
Keytab generation during Puppet run: Implementation
exec{}
POST /catalog
Manifests
Hiera
Compiler
Templates
Send factsReceive catalog
Apply catalog
ERB template<%= `kadmin -w S3cr3tPass ...` %>
Task: Generate keytab before/during catalog compilation
file { "/etc/krb5.keytab": ... content => template("krb5/keytab.erb")}
Keytab generation during Puppet run: Implementation
exec{}
POST /catalog
Manifests
Hiera
Compiler
Templates
Send factsReceive catalog
Apply catalog
ERB template<%= `kadmin -w S3cr3tPass ...` %>
Puppet generate( ) function
Custom Puppet function
Part of agent certificate signing
Task: Generate keytab before/during catalog compilation
Our implementation: Custom function
PSEUDO CODE
$h_keytab = hiera('krb5-keytab', '*undefined*')
if ($h_keytab == '*undefined*') { $keytab = generate_keytab( { options... } )} else { $keytab = $h_keytab}
file { "/etc/krb5.keytab": replace => true, content => $keytab,}
POST /catalog
Manifests
Hiera
Compiler
Templates
Send factsReceive catalog
Apply catalog$h_keytab = hiera('krb5-keytab', generate_keytab(...))
Our implementation: Custom function
$h_keytab = hiera('krb5-keytab', '*undefined*')
if ($h_keytab == '*undefined*') { $keytab = generate_keytab(...)} else { $keytab = $h_keytab}
file { "/etc/krb5.keytab": replace => true, content => $keytab,}
● Puppet master has admin keytab
Our Implementation Details:
● We store the admin keytab in hiera for easier packaging
Our implementation: Custom function
● Puppet master knows admin keytab
Our Implementation Details:$h_keytab = hiera('krb5-keytab', '*undefined*')
if ($h_keytab == '*undefined*') { $keytab = generate_keytab(...) store_keytab_in_hiera(...)} else { $keytab = $h_keytab}
file { "/etc/krb5.keytab": replace => true, content => $keytab,}
● We store the admin keytab in hiera for easier packaging
● Call a custom function to store the generated keytab in hiera
Our implementation: Custom function
● Puppet master knows admin keytab
Our Implementation Details:$h_keytab = hiera('krb5-keytab', '*undefined*')
if ($h_keytab == '*undefined*') { $keytab = generate_keytab(...) store_keytab_in_hiera(...)} else { $keytab = base64('decode',$h_keytab)}
file { "/etc/krb5.keytab": replace => true, content => $keytab,}
● We store the admin keytab in hiera for easier packaging
● Call a custom function to store the generated keytab in hiera
● Base64 encode keytabs in hiera; decode them when retrieved
Scalability challengeData Center #1 Data Center #2
Puppet Master1A
Puppet Master1B
Puppet Master2A
Puppet Master2B
Node Node Node Node Node Node
CouchDB
CouchDB
CouchDB
CouchDB
Scalability challengeData Center #1
Puppet Master1A
Puppet Master1B
Node Node Node
CouchDB
CouchDB
CouchDB Pros:
● Open source (Apache)
● Native multi-master replication
● Simple HTTP/JSON interface
● Records are versioned
● Web GUI (futon)
CouchDB Cons:
● Renew replication at restart
● No LDAP auth for users
Other use casesCertificate signing for websites
• Generate key and CSRvia script on Puppet Master
• Sign certificate with the CA's private key from Puppet Master
• Store key and certificate in hiera
• Use file{} to install certificates
• Notify service{} to restart web server to pick up new certificate
LDAP server configuration
• Use package{}, service{} etc.in typical fashion
• Use file{} to install configuration files on LDAP server
• Add replication agreement using LDAP admin credential (can run from Puppet Master)
• Notify service{} to restartwhen replication is set up
ResourcesPuppet module:https://forge.puppetlabs.com/collectivemedia/krb5keytabhttps://github.com/collectivemedia/puppet-krb5keytab
Slide deck:https://slideshare.net/kpaulisse/nyc-puppetpresentation
Collective website:
http://www.collective.com
Collective careers:
Contact info:
Kevin Paulisse
Work: [email protected]
Home: [email protected]
Twitter: @kpaulisse / Github: kpaulisse
Recommended
