Puppet as data_chicago

Dan Bode| Puppet Labs

dan@puppetlabs.com bodepd <on> [twitter,freenode]

Puppet as DataTransformations

# puppetcamp # puppetize @ puppetlabs

What is it about?

• Deconstructing Puppet to data

• Why you should care

Dissecting a Puppet Run

Part One

# puppetconf # puppetize @ puppetlabs

Facter, who am I?

Agent

Hi! your facts are:

kernel=linuxipaddress=10.0.0.3macaddress=…

# puppetconf # puppetize @ puppetlabs

Hi Mr. Master, I need a catalog. Here are my facts

http://www.dgcomputers.org/testimonials.php

Agent

facts

# puppetconf # puppetize @ puppetlabs

facts

AgentThanks for you

facts.I’ll just store them

in PuppetDBPuppetDB

# puppetconf # puppetize @ puppetlabs

PuppetDBPuppetDB

facts

ENCENC

Agent

Mr. ENC, is this host defined as

an external node?Yep, he should be an apache server.

Here is the definition

nodes

# puppetconf # puppetize @ puppetlabs

PuppetDBPuppetDB

facts

AgentJust compiled

your catalog. One sec while I store it

in PuppetDB.

catalog

# puppetconf # puppetize @ puppetlabs

PuppetDBPuppetDB

facts

Agent

catalog

Here is your catalog. Send me a report and let me know how it

went!

catalog

# puppetconf # puppetize @ puppetlabs

PuppetDBPuppetDB

facts

Agent

catalog

I hate to be a bother, but

can you compute the md5sums of

this file?

catalog

# puppetconf # puppetize @ puppetlabs

PuppetDBPuppetDB

facts

Agent

catalog

Oh, I need that one!

Can I get the file.

catalog

# puppetconf # puppetize @ puppetlabs

PuppetDBPuppetDB

facts

Agent

catalog

Thanks, now let’s just do that 1000

more times.

catalog

# puppetconf # puppetize @ puppetlabs

PuppetDBPuppetDB

facts

Agent

catalog

You know ‘content’ will embed the contents in the catalog?

catalog

# puppetconf # puppetize @ puppetlabs

PuppetDBPuppetDB

facts

AgentJust finished

applying. Here are the results.

report

catalog

Puppet’s internal data language (indirectory/termini)

Part Two

# puppetconf # puppetize @ puppetlabs

facts find from terminus facter

Agent

# puppetconf # puppetize @ puppetlabs

catalog find from terminus rest

http://www.dgcomputers.org/testimonials.php

Agent

facts

# puppetconf # puppetize @ puppetlabs

facts

Agentfacts save to

terminus puppetdb

PuppetDB

# puppetconf # puppetize @ puppetlabs

PuppetDBPuppetDB

facts

ENCENC

Agent

node find from terminus exec (or

ldap)

nodes

# puppetconf # puppetize @ puppetlabs

PuppetDBPuppetDB

facts

Agentcatalog find from terminus compiler

catalog

# puppetconf # puppetize @ puppetlabs

PuppetDBPuppetDB

facts

Agentcatalog save to

terminus puppetdb

catalog

# puppetconf # puppetize @ puppetlabs

FacterFacter ENCENC

Disecting a Puppet Run

Com

pil

er

Com

pil

er

Config Catalogs

Nodes/Manife

st

Reports

Facts

# puppetconf # puppetize @ puppetlabs

CLI commands

# puppetcamp # puppetize @ puppetlabs

CLI Puppet Facts

# mkdir –p /tmp/yaml/facts

# puppet facts find node_name --render-as yaml \

> /tmp/yaml/facts/node_name.yaml

# puppetcamp # puppetize @ puppetlabs

Creating a node (optional):

# mkdir –p /tmp/yaml/nodes

# puppet node find node_name \

--node_terminus=exec \

--external_nodes=/etc/puppet/nodes.sh \

--facts_terminus=yaml \

--clientyamldir=/tmp/yaml/ --render-as=yaml \

> /tmp/yaml/nodes/node_name.yaml

# puppetcamp # puppetize @ puppetlabs

Applying a catalog:

# puppet catalog find node_name \

/tmp/catalog.yaml

# puppetcamp # puppetize @ puppetlabs

Creating a catalog:

# puppet apply –catalog /tmp/catalog.yaml

(its easy assuming you are not using file sources)

# puppetconf # puppetize @ puppetlabs

Fun with IRB

# puppetcamp # puppetize @ puppetlabs

IRB Facts

irb:> require ‘puppet/face’

> Puppet.parse_config # required Puppet > 3.0

> facts=Puppet::Face[:facts, :current].find('node_name')

# puppetcamp # puppetize @ puppetlabs

Access a Fact value (irb):

…

> facts.values['ipaddress']

=> "10.0.2.15"

# puppetcamp # puppetize @ puppetlabs

Creating a node (from irb):

…

> node=Puppet::Node.new('node_name',

{:classes => {:foo => {:bar => :baz}}})

>node.merge(facts.values)

# puppetcamp # puppetize @ puppetlabs

Creating a catalog:

…

irb> catalog=Puppet::Face[:catalog, :current].\

find('node_name', :extra => { :node => node})

Interacting with Puppet’s Data

Use Cases

# puppetcamp # puppetize @ puppetlabs

Inspecting the catalog:

• What types are in the catalog?

irb> catalog.resources.collect {|r| r.type }.uniq

• Gimme a resource:

irb>catalog.resource(‘Package[httpd]’)

# puppetcamp # puppetize @ puppetlabs

Rspec Puppet:

let :facts do

{:operatingsystem => ‘Redhat’}

end

let :params do

{:bind_address => ‘0.0.0.0’

end

it { should contain_file(‘/tmp/foo.conf’) }

# puppetcamp # puppetize @ puppetlabs

Thundering Herd

Pre-compile catalogs for faster auto-scaling

# puppetcamp # puppetize @ puppetlabs

Applying pre-compiled catalogs:

• Gather facts from an ec2 instance

• Generate a single catalog

• Apply that catalog to all hosts

puppet apply --catalog /tmp/catalog.json –

server puppet-fileserver

# puppetcamp # puppetize @ puppetlabs

DMZ

tcp over USB

Interacting with Puppet’s Data

Use Cases

# puppetcamp # puppetize @ puppetlabs

Hacking reports

Everything in Puppet is a state transition

User[‘dan’] : absent -> present

User[‘dan’][‘shell’] -> ‘/sbin/nologin’ -> /bin/bash

# puppetcamp # puppetize @ puppetlabs

Setting up the agent:

[agent]

report=true

# puppetcamp # puppetize @ puppetlabs

Archive reports in your yamldir

[master]

reports = store

# puppetcamp # puppetize @ puppetlabs

Puppet reports

$ cd `puppet config print reportdir`

$ ls

node1 node2 node3

$ ls node1

# puppetcamp # puppetize @ puppetlabs

Every report from every run ever

$ ls node1

201206060256.yaml 201206060303.yaml 201206060519.yaml 201206122349.yaml 201206122354.yaml 201206130002.yaml

# puppetcamp # puppetize @ puppetlabs

Lets crack one open!

Irb > require ‘yaml’

>reports=YAML.load_file('201206130002.yaml')

# puppetcamp # puppetize @ puppetlabs

Have a look

>(reports.methods - Object.methods).sort

Notice the following methods:

# puppetcamp # puppetize @ puppetlabs

High level data

> reports.exit_status

0

> reports.status

=> "unchanged"

> reports.host

”node1”

# puppetcamp # puppetize @ puppetlabs

metrics

> reports.metrics.keys

["resources", "events", "changes", "time"]

> reports.metrics['resources']

[‘failed’, 0],[ ‘changed’, ‘7’]

# puppetcamp # puppetize @ puppetlabs

And the awesome sauce

> reports.resource_statuses.keys

=> ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe-puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]

# puppetcamp # puppetize @ puppetlabs

And the awesome sauce

> status = reports.resource_statuses

> status.keys

=> ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe-puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]

# puppetcamp # puppetize @ puppetlabs

And the awesome sauce>events = status["File[/etc/swift/swift.conf]"].events

> events.first.status

"success”

> events.first.desired_value

:present

> events.first.previous_value

=> :absent

amp

Thank YouDan Bode| Puppet Labs dan@puppetlabs.com