2014-10-28 PuppetNYC meetup: Managing Externally Generated Files with Puppet

Managing ExternallyGenerated Files with Puppet

Kevin PaulisseDevOps Engineer and Team Lead

Collective, Inc.

Hello, I am Kevin Paulisse

Background:

• Linux guy for 17+ years, Puppet devotee for 3+ years• Current position: DevOps engineer and team lead @ Collective• Prior: Shopbop (Amazon), Philips, State of WI, TDS, Alliant Energy

Contact information:

• Work e-mail: kpaulisse@collective.com• Personal e-mail: kpaulisse@gmail.com• Twitter: @kpaulisse• Github: kpaulisse• LinkedIn: https://www.linkedin.com/in/kpaulisse

kpaulisse@collective.comkpaulisse@gmail.com@kpaulissehttps://github.com/kpaulissehttps://linkedin.com/in/kpaulisse

About Collective

What Collective does:

• Serve Ads: 10's of thousands of requests/sec• Bid on Ads: 100's of thousands of requests/sec• Data Science: 1000's of predictive models weekly

Collective's technology stack:

• Linux (physical & virtual)• Puppet and Foreman• 3 new data centers this year• 300 node Hadoop cluster

Collective is hiring:

• DevOps/Linux Engineers• Data Scientists, Devs, etc.

recruiting@collective.comhttp://www.collective.com

Agenda

• Introduction

• Kerberos crash course

• Managing Kerberos keytab files with Puppet:

• Traditional approaches: file{} and exec{}

• Other approaches to generating keytabs

• Our implementation

• Scaling the solution to multiple Puppet masters

• Other use cases

Kerberos crash course: Terminology

Kerberos: Networkauthentication protocol

Principal: An account

Keytab: A file containingencrypted keys forprincipals

Host Keytab: On Linux,/etc/krb5.keytab

Kerberos crash course: Keytab files

Each host needs its own Kerberos principal and corresponding keytab.

A Kerberos Administrator creates and manipulates principals.

Task NeedsAdmin

ScramblesPassword

Determine if there is a Kerberos principal for the host

Create the Kerberos principal if none exists

Save Kerberos credentials to a keytab file

(Scrambling password refers to MIT Kerberos -- other systems may differ.)

Installing Kerberos keytab with file{}

file { "/etc/krb5.keytab": ensure => present, owner => "root", group => "root", mode => 0400, source => WhatGoesHere?}

● The keytab file must already exist somewhere

● The above resource does not generate the file

Installing Kerberos keytab with exec{}

exec { "create-etc-krb5.keytab": command => "kadmin -w S3cr3tPass ...", creates => "/etc/krb5.keytab", user => "root",}

● The exec resource runs on the agent

● Each agent gets to see your administrative credential

Generate keytab asynchronously: Design

Keytab generation

● Use the admin credential on just one server

● Generate keytab before Puppet runs (put keytab where Puppet can find it)

● Store the keytabs in such a way that a host can see only its own keytab

Puppet manifest

● Use a file{} resource to define content/source

● Host keytab in catalog (not admin credentials)

● Usual file{} features: set permissions, restart services, etc.

Generate keytab asynchronously: Implementation

Manual processDocs: "Don't forget to run the keytab script"

Management system (e.g. The Foreman)Generate keytab via a "hook" from management system when a host is added to the configurationor a Puppet run is scheduled

Task: Generate a keytab file before the Puppet run

Generate keytab during Puppet run: Design

Keytab generation

● Keytab has to be generated during the Puppet run

● Keytab has to be available on Puppet master so it can be included in the catalog

● Don't use exec{} on agent to generate keytab

Puppet agent

● Use a file{} resource to define content/source

● Host keytab in catalog (not admin credentials)

● Usual file{} features: set permissions, restart services, etc.

Internals of a Puppet run

Read Further: https://docs.puppetlabs.com/puppet/3.7/reference/subsystem_agent_master_comm.html

1. SSL certificate generation and signing

2. Plugin sync

3. Build and fetch catalog (pictured at left)

a. HTTPS POST with facts

b. Catalog = everything but file sources

4. Apply (and fetch files where needed)

5. Send reportPOST /catalog

Manifests

Hiera

Compiler

Templates

Send factsReceive catalog

Apply catalog

Keytab generation during Puppet run: Implementation

exec{}

POST /catalog

Manifests

Hiera

Compiler

Templates

Send factsReceive catalog

Apply catalog

ERB template<%= `kadmin -w S3cr3tPass ...` %>

Task: Generate keytab before/during catalog compilation

file { "/etc/krb5.keytab": ... content => template("krb5/keytab.erb")}

Keytab generation during Puppet run: Implementation

exec{}

POST /catalog

Manifests

Hiera

Compiler

Templates

Send factsReceive catalog

Apply catalog

ERB template<%= `kadmin -w S3cr3tPass ...` %>

Puppet generate( ) function

Custom Puppet function

Part of agent certificate signing

Task: Generate keytab before/during catalog compilation

Our implementation: Custom function

PSEUDO CODE

$h_keytab = hiera('krb5-keytab', '*undefined*')

if ($h_keytab == '*undefined*') { $keytab = generate_keytab( { options... } )} else { $keytab = $h_keytab}

file { "/etc/krb5.keytab": replace => true, content => $keytab,}

POST /catalog

Manifests

Hiera

Compiler

Templates

Send factsReceive catalog

Apply catalog$h_keytab = hiera('krb5-keytab', generate_keytab(...))

Our implementation: Custom function

$h_keytab = hiera('krb5-keytab', '*undefined*')

if ($h_keytab == '*undefined*') { $keytab = generate_keytab(...)} else { $keytab = $h_keytab}

file { "/etc/krb5.keytab": replace => true, content => $keytab,}

● Puppet master has admin keytab

Our Implementation Details:

● We store the admin keytab in hiera for easier packaging

Our implementation: Custom function

● Puppet master knows admin keytab

Our Implementation Details:$h_keytab = hiera('krb5-keytab', '*undefined*')

if ($h_keytab == '*undefined*') { $keytab = generate_keytab(...) store_keytab_in_hiera(...)} else { $keytab = $h_keytab}

file { "/etc/krb5.keytab": replace => true, content => $keytab,}

● We store the admin keytab in hiera for easier packaging

● Call a custom function to store the generated keytab in hiera

Our implementation: Custom function

● Puppet master knows admin keytab

Our Implementation Details:$h_keytab = hiera('krb5-keytab', '*undefined*')

if ($h_keytab == '*undefined*') { $keytab = generate_keytab(...) store_keytab_in_hiera(...)} else { $keytab = base64('decode',$h_keytab)}

file { "/etc/krb5.keytab": replace => true, content => $keytab,}

● We store the admin keytab in hiera for easier packaging

● Call a custom function to store the generated keytab in hiera

● Base64 encode keytabs in hiera; decode them when retrieved

Scalability challengeData Center #1 Data Center #2

Puppet Master1A

Puppet Master1B

Puppet Master2A

Puppet Master2B

Node Node Node Node Node Node

CouchDB

CouchDB

CouchDB

CouchDB

Scalability challengeData Center #1

Puppet Master1A

Puppet Master1B

Node Node Node

CouchDB

CouchDB

CouchDB Pros:

● Open source (Apache)

● Native multi-master replication

● Simple HTTP/JSON interface

● Records are versioned

● Web GUI (futon)

CouchDB Cons:

● Renew replication at restart

● No LDAP auth for users

Other use casesCertificate signing for websites

• Generate key and CSRvia script on Puppet Master

• Sign certificate with the CA's private key from Puppet Master

• Store key and certificate in hiera

• Use file{} to install certificates

• Notify service{} to restart web server to pick up new certificate

LDAP server configuration

• Use package{}, service{} etc.in typical fashion

• Use file{} to install configuration files on LDAP server

• Add replication agreement using LDAP admin credential (can run from Puppet Master)

• Notify service{} to restartwhen replication is set up

ResourcesPuppet module:https://forge.puppetlabs.com/collectivemedia/krb5keytabhttps://github.com/collectivemedia/puppet-krb5keytab

Slide deck:https://slideshare.net/kpaulisse/nyc-puppetpresentation

Collective website:

http://www.collective.com

Collective careers:

recruiting@collective.com

Contact info:

Kevin Paulisse

Work: kpaulisse@collective.com

Home: kpaulisse@gmail.com

Twitter: @kpaulisse / Github: kpaulisse