Puppet as data_chicago

Dan Bode| Puppet Labs

[email protected] bodepd <on> [twitter,freenode]

Puppet as DataTransformations

# puppetcamp # puppetize @ puppetlabs

What is it about?

• Deconstructing Puppet to data

• Why you should care

Dissecting a Puppet Run

Part One

# puppetconf # puppetize @ puppetlabs

Facter, who am I?

Agent

Hi! your facts are:

kernel=linuxipaddress=10.0.0.3macaddress=…


Hi Mr. Master, I need a catalog. Here are my facts

http://www.dgcomputers.org/testimonials.php

Agent

facts


facts

AgentThanks for you

facts.I’ll just store them

in PuppetDBPuppetDB


PuppetDBPuppetDB

facts

ENCENC

Agent

Mr. ENC, is this host defined as

an external node?Yep, he should be an apache server.

Here is the definition

nodes


PuppetDBPuppetDB

facts

AgentJust compiled

your catalog. One sec while I store it

in PuppetDB.

catalog


PuppetDBPuppetDB

facts

Agent

catalog

Here is your catalog. Send me a report and let me know how it

went!

catalog


PuppetDBPuppetDB

facts

Agent

catalog

I hate to be a bother, but

can you compute the md5sums of

this file?

catalog


PuppetDBPuppetDB

facts

Agent

catalog

Oh, I need that one!

Can I get the file.

catalog


PuppetDBPuppetDB

facts

Agent

catalog

Thanks, now let’s just do that 1000

more times.

catalog


PuppetDBPuppetDB

facts

Agent

catalog

You know ‘content’ will embed the contents in the catalog?

catalog


PuppetDBPuppetDB

facts

AgentJust finished

applying. Here are the results.

report

catalog

Puppet’s internal data language (indirectory/termini)

Part Two


facts find from terminus facter

Agent


catalog find from terminus rest

http://www.dgcomputers.org/testimonials.php

Agent

facts


facts

Agentfacts save to

terminus puppetdb

PuppetDB


PuppetDBPuppetDB

facts

ENCENC

Agent

node find from terminus exec (or

ldap)

nodes


PuppetDBPuppetDB

facts

Agentcatalog find from terminus compiler

catalog


PuppetDBPuppetDB

facts

Agentcatalog save to

terminus puppetdb

catalog


FacterFacter ENCENC

Disecting a Puppet Run

Com

pil

er

Com

pil

er

Config Catalogs

Nodes/Manife

st

Reports

Facts


CLI commands


CLI Puppet Facts

# mkdir –p /tmp/yaml/facts

# puppet facts find node_name --render-as yaml \

> /tmp/yaml/facts/node_name.yaml


Creating a node (optional):

# mkdir –p /tmp/yaml/nodes

# puppet node find node_name \

--node_terminus=exec \

--external_nodes=/etc/puppet/nodes.sh \

--facts_terminus=yaml \

--clientyamldir=/tmp/yaml/ --render-as=yaml \

> /tmp/yaml/nodes/node_name.yaml


Applying a catalog:

# puppet catalog find node_name \

/tmp/catalog.yaml


Creating a catalog:

# puppet apply –catalog /tmp/catalog.yaml

(its easy assuming you are not using file sources)


Fun with IRB


IRB Facts

irb:> require ‘puppet/face’

> Puppet.parse_config # required Puppet > 3.0

> facts=Puppet::Face[:facts, :current].find('node_name')


Access a Fact value (irb):

…

> facts.values['ipaddress']

=> "10.0.2.15"


Creating a node (from irb):

…

> node=Puppet::Node.new('node_name',

{:classes => {:foo => {:bar => :baz}}})

>node.merge(facts.values)


Creating a catalog:

…

irb> catalog=Puppet::Face[:catalog, :current].\

find('node_name', :extra => { :node => node})

Interacting with Puppet’s Data

Use Cases


Inspecting the catalog:

• What types are in the catalog?

irb> catalog.resources.collect {|r| r.type }.uniq

• Gimme a resource:

irb>catalog.resource(‘Package[httpd]’)


Rspec Puppet:

let :facts do

{:operatingsystem => ‘Redhat’}

end

let :params do

{:bind_address => ‘0.0.0.0’

end

it { should contain_file(‘/tmp/foo.conf’) }


Thundering Herd

Pre-compile catalogs for faster auto-scaling


Applying pre-compiled catalogs:

• Gather facts from an ec2 instance

• Generate a single catalog

• Apply that catalog to all hosts

puppet apply --catalog /tmp/catalog.json –

server puppet-fileserver


DMZ

tcp over USB

Interacting with Puppet’s Data

Use Cases


Hacking reports

Everything in Puppet is a state transition

User[‘dan’] : absent -> present

User[‘dan’][‘shell’] -> ‘/sbin/nologin’ -> /bin/bash


Setting up the agent:

[agent]

report=true


Archive reports in your yamldir

[master]

reports = store


Puppet reports

$ cd `puppet config print reportdir`

$ ls

node1 node2 node3

$ ls node1


Every report from every run ever

$ ls node1

201206060256.yaml 201206060303.yaml 201206060519.yaml 201206122349.yaml 201206122354.yaml 201206130002.yaml


Lets crack one open!

Irb > require ‘yaml’

>reports=YAML.load_file('201206130002.yaml')


Have a look

>(reports.methods - Object.methods).sort

Notice the following methods:


High level data

> reports.exit_status

0

> reports.status

=> "unchanged"

> reports.host

”node1”


metrics

> reports.metrics.keys

["resources", "events", "changes", "time"]

> reports.metrics['resources']

[‘failed’, 0],[ ‘changed’, ‘7’]


And the awesome sauce

> reports.resource_statuses.keys

=> ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe-puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]


And the awesome sauce

> status = reports.resource_statuses

> status.keys

=> ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe-puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]


And the awesome sauce>events = status["File[/etc/swift/swift.conf]"].events

> events.first.status

"success”

> events.first.desired_value

:present

> events.first.previous_value

=> :absent

amp

Thank YouDan Bode| Puppet Labs [email protected]

Documents

Puppet as data_chicago