Introduction to Chef

Introduction to Infrastructure as Code & Automation / Introduction to Chef Ned Harris, Solution Architect Chef

2http://www.flickr.com/photos/michaelheiss/3090102907/

Complexity

Managing Complexity

•  SSH, make with the typey typey

Managing Complexity

•  SSH, make with the typey typey •  Keep notes in ~/server.txt

Managing Complexity

•  SSH, make with the typey typey •  Keep notes in ~/server.txt •  Move notes to the wiki

Managing Complexity

•  SSH, make with the typey typey •  Keep notes in ~/server.txt •  Move notes to the wiki •  Custom scripts (setup.sh)

Managing Complexity

•  SSH, make with the typey typey •  Keep notes in ~/server.txt •  Move notes to the wiki •  Custom scripts (setup.sh) •  Golden Images

Golden Images are not the answer

• Gold is heavy

• Hard to transport

• Hard to mold

• Easy to lose configuration detail

http://www.flickr.com/photos/garysoup/2977173063/

Typical Infrastructure

Memcache

Postgres Slaves

Postgres Master

Nagios Graphite

Jboss App

New Compliance Mandate!

Jboss App

Memcache

Postgres Slaves

Postgres Master

Nagios Graphite

• Move SSH off port 22

•  Lets put it on 2022

6 Golden Image Updates

Jboss App

Memcache

Postgres Slaves

Postgres Master

Nagios Graphite

•  edit /etc/ssh/sshd_config

1 2

3

4

5

6

12 Instance Replacements

Jboss App

Memcache

Postgres Slaves

Postgres Master

Nagios Graphite

•  Delete, launch

1 2

3 4 5 6 7

8 9

10 11

12

•  Repeat

•  Typically manually

Done in Maintenance Windows

•  Don’t break anything!

•  Bob just got fired =(

5

Jboss App

Memcache

Postgres Slaves

Postgres Master

Nagios Graphite 1 2

4 5 6 7

8 9

10 11

12

3

12

Different IP Addresses?

Jboss App

Memcache

Postgres Slaves

Postgres Master

Nagios Graphite

•  Invalid configs!

Managing Complexity

•  SSH, make with the typey typey •  Keep notes in ~/server.txt •  Move notes to the wiki •  Custom scripts (setup.sh) •  Golden Images •  Policy-driven configuration management

Policies

•  Declarations about the state of thing in a system •  applied repeatedly and repair the system when needed •  often change

Repeatable Operations

•  Idempotent •  can be applied an infinite number of times and yield the same result every time

•  Convergent •  test state and repair if needed

18

Policy Evolves

Following Policy

•  A control loop keeps the system stable and allows for change when policy is updated

20

Policy Evolves That's great and all, !but tell me about !Chef! !

What is Chef?

•  Framework for managing complexity •  Infrastructure as code

•  a domain-specific language (DSL) for describing convergent operations

•  A community of professionals •  A company

The Chef Software Platform

Chef Development Kit

Cookbook and Policy Authoring

Test-Driven Infrastructure

Chef Server

Management Console

Analytics Platform

High Availability and Replication

Chef Client Nodes

Data Center

The Cloud

How does Chef work?

•  Ensure desired state by continually testing and repairing individual resources in the system

•  Compose policies using a series of abstractions

Desired Configuration

Node

Chef Server What policy should I follow?


Node


"recipe[ntp::client]" "recipe[users]" "role[webserver]"



"recipe[ntp::client]" "recipe[users]" "role[webserver]"

Recipes

package "apache2"resource one

Recipes

package "apache2"

template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode "0644" variables(:allow_override => "All") notifies :reload, "service[apache2]"end

resource one

resource two

Recipes

package "apache2"

template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode "0644" variables(:allow_override => "All") notifies :reload, "service[apache2]"end

service "apache2" do action [:enable,:start] supports :reload => trueend

resource one

resource two

resource three

Recipes

resource one

Recipes

resource one

resource two

Recipes

resource one

resource two

resource three

Built-in Resources

• package • template • service • cron • directory • mount

• user • group •  registry_key •  remote_directory •  route •  ...and many more!

Managing Complexity

•  Organizations •  Environments •  Roles •  Nodes •  Recipes •  Cookbooks •  Search •  Data

Their Infrastructure

Organizations

My Infrastructure Your Infrastructure

Environments

Development Staging Production

Roles

Load Balancers

Application Servers

DB Cache

Database

Nodes

40

Search

•  Search for nodes with Roles •  Find Topology Data

•  IP addresses •  Hostnames •  FQDNs

http://www.flickr.com/photos/kathycsus/2686772625 40

Search for Nodes pool_members = search("node","role:webserver") template "/etc/haproxy/haproxy.cfg" do source "haproxy-‐app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end

Webservers

HAProxy Configuration

HA Proxy

Webservers

HAProxy Load Balancer

HA Proxy Enterprise Chef

pool_members = search("node","role:webserver")

Webservers



Webservers?


Webservers



Webservers?


Webservers



Webservers?


{ "web01" : { "hostname" : "web01", "ipaddress" : "10.1.1.1" }, "web02" : { "hostname" : "web02", "ipaddress" : "10.1.1.2" }, "web03" : { "hostname" : "web03", "ipaddress" : "10.1.1.3" }, "web04" : { "hostname" : "web04", "ipaddress" : "10.1.1.4" }, "web05" : { "hostname" : "web05", "ipaddress" : "10.1.1.5" }, "web06" : { "hostname" : "web06", "ipaddress" : "10.1.1.6" } }

Webservers



Webservers?



Webservers



Webservers?



pool_members{ "web01" : { "hostname" : "web01", "ipaddress" : "10.1.1.1" }, "web02" : { "hostname" : "web02", "ipaddress" : "10.1.1.2" }, "web03" : { "hostname" : "web03", "ipaddress" : "10.1.1.3" }, "web04" : { "hostname" : "web04", "ipaddress" : "10.1.1.4" }, "web05" : { "hostname" : "web05", "ipaddress" : "10.1.1.5" }, "web06" : { "hostname" : "web06", "ipaddress" : "10.1.1.6" } }

Search for Nodes pool_members = search("node","role:webserver") template "/etc/haproxy/haproxy.cfg" do source "haproxy-‐app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end

Pass results into Templates # Set up application listeners here. listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -‐%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -‐%> <% if node["haproxy"]["enable_admin"] -‐%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -‐%>

Webservers


HA Proxy pool_members{ "web01" : { "hostname" : "web01", "ipaddress" : "10.1.1.1" }, "web02" : { "hostname" : "web02", "ipaddress" : "10.1.1.2" }, "web03" : { "hostname" : "web03", "ipaddress" : "10.1.1.3" }, "web04" : { "hostname" : "web04", "ipaddress" : "10.1.1.4" }, "web05" : { "hostname" : "web05", "ipaddress" : "10.1.1.5" }, "web06" : { "hostname" : "web06", "ipaddress" : "10.1.1.6" } }

<% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%>

Webservers


HA Proxy



haproxy.cfgserver web01 10.1.1.1 weight 1 maxconn 1 check

Webservers


HA Proxy




server web02 10.1.1.2 weight 1 maxconn 1 check

Webservers


HA Proxy






Webservers


HA Proxy







Webservers


HA Proxy








Webservers


HA Proxy









So when this…

Jboss App

Memcache

Postgres Slaves

Postgres Master

Nagios Graphite

…becomes this…

Jboss App

Memcache

Postgres Slaves

Postgres Master

Nagios Graphite

…this can happen automaticaly!

Memcache

Postgres Slaves

Postgres Master

Nagios Graphite

Jboss App

Memcache

Postgres Slaves

Postgres Master

Nagios Graphite

Count the resources

Nagios Graphite

Nagios Graphite

Memcache

Postgres Slaves

•  Load balancer config

•  Nagios host ping

•  Nagios host ssh

•  Nagios host HTTP

•  Nagios host app health

•  Graphite CPU

•  Graphite Memory

•  Graphite Disk

•  Graphite SNMP

•  Memcache firewall

•  Postgres firewall

•  Postgres authZ config

•  12+ resource changes for 1 node addition

Jboss App

Build Anything

• Simple internal applications • Complex external applications • Workstations • Hadoop clusters •  IaaS infrastructure • PaaS infrastructure • SaaS applications • Storage systems • You name it

http://www.flickr.com/photos/hyku/245010680/

And Manage it Simply

• Automatically reconfigure everything

• Linux, Windows, Unixes, BSDs

• Load balancers • Metrics collection systems • Monitoring systems • Cloud migrations become

trivial http://www.flickr.com/photos/helico/404640681/

ChefDK

Increase CHEF adoption through ChefDK

●  ChefDK: CHEF Software Development Kit, fully supported with the Chef Premium Subscription

●  Workflow Definition: Our recommendation on the process to test and verify your infrastructure code before committing it to source control and shipping it to production.

●  Workflow Enhancement: Based on customer feedback and use cases

ChefDK: In the Box

First Class Support on Windows, Linux, and OSX for the entire suite of Chef development tools

●  Test Kitchen: Virtualized testing harness ●  Berkshelf: Dependency solver ●  Chef-Vault: Secrets management ●  Rubocop / Foodcritic: Code linting ●  Chefspec: In-memory Unit Testing ●  Chef.bin: New wrapper binary to tie it all

together, with new extensible cookbook generators.

The Chef workflow

Create new skeleton cookbook.

Create a VM environment for cookbook development.

Write/debug cookbook recipes (iterative step).

Perform acceptance tests.

Deploy to production.

Questions?

Technology

Introduction to Chef