Chef Cookbook Testing and Continuous Integration

Chef Cookbook Testing and Continuous IntegrationJulian C. DunnSenior Consultant, Opscode, Inc.<[email protected]>

Friday, June 14, 13

mailto:[email protected]


Why Write Cookbook Tests?

• Catch development errors• Catch regressions• Code to requirements (TDD, BDD)• Smaller batch sizes == lower blast radius• Continuous delivery of cookbooks!• Build confidence that your infrastructure is

working correctly

Friday, June 14, 13

Software Development Lifecycle for Sysadmins

Friday, June 14, 13

Devops Is A Two-Way Street• It’s great when

developers care about• uptime!• scaling!• deployment!

• Put them on call! etc. etc. etc.

Friday, June 14, 13

Devops Is A Two-Way Street

• Systems Administrators also have as much or more to learn from developers as well!

Friday, June 14, 13

Source Code

Compiler

Artifact

Test

Typical Development Workflow

• Typical Software Development Workflow:

• Write Source

• Compile Source

• Deploy Artifacts

• Write Tests

• Run Tests

Friday, June 14, 13

Source Code

Compiler

Artifact

Test


• Source Code

• The recipe for a computer program

• Edited directly

• Managed via Source Control Software

Friday, June 14, 13

Source Code

Compiler

Artifact

Test


• Compiler

• Takes source code and converts it into executable programs

Friday, June 14, 13

Source Code

Compiler

Artifact

Test


• Artifact

• Artifacts are executable programs created by compilers.

• Compiled artifacts cannot be edited directly. Source code must be changed and re-compiled to produce a new build artifact.

Friday, June 14, 13

Source Code

Compiler

Artifact

Test


• Testing

• Write tests to verify that code works as intended

• Run tests at different stages of the code lifecycle to ensure correctness

Friday, June 14, 13

Source Code

Compiler

Artifact

Test


• When developing software, most time isn’t actually spent coding

• 10-second changes to source code can take minutes to vet

• Compiling code

• Deploying code

• Writing & Running tests

Friday, June 14, 13

Friday, June 14, 13

Source Code

Compiler

Artifact

Test

Typical Development WorkflowFun!

Boring!

• Things that are fun:

• Designing programs!

• Writing source code!

• Things that are boring:

• Compiling code

• Deploying artifacts

• Running tests

Friday, June 14, 13

Look familiar to any Chefs?Friday, June 14, 13

Worksta(on:+Knife+cookbook+

create+Worksta(on:+Edit+cookbook+


upload+

Provision+target+Bootstrap+target+

Worksta(on:+Edit+target+run+

list+

ssh+target+ Target:+Run+chef>client+

Traditional Cookbook Development: First Chef Run

Friday, June 14, 13

Worksta(on:+edit+cookbook+

Worksta(on:+knife+cookbook+

upload+

Target:+Run+chef9client+

Traditional Cookbook Development: Subsequent Runs

Friday, June 14, 13

Source'Code'

Compiler'

Ar/fact'

Test'

Boooooooring• Too much time doing

“paperwork”• vi recipes/something.rb

• knife cookbook upload

• sudo pkill -USR1 chef-client

• #%$#%$ something broke, let me do that all again

• Not enough time doing fun stuff!

• Writing recipes

Less Fun!

More Boring!

X

Friday, June 14, 13

Source'Code'

Compiler'

Ar/fact'

Test'

Boooooooring• Too much time doing

“paperwork”• vi recipes/something.rb

• knife cookbook upload

• sudo pkill -USR1 chef-client

• #%$#%$ something broke, let me do that all again

• Not enough time doing fun stuff!

• Writing recipes

Less Fun!

More Boring!

XTHIS

SUCKS!

Friday, June 14, 13

Chefs on a Plane

What if...

Friday, June 14, 13

Chefs on a Plane

What if...Worksta(on:+Knife+cookbook+



upload+



list+


Target:(Run(chef/client(

Worksta6on:(edit(cookbook(

Worksta6on:(knife(cookbook(

upload(

...we could automate all

of this...

Friday, June 14, 13

Chefs on a Plane

What if...Worksta(on:+Knife+cookbook+



upload+



list+





upload(


of this...

...to run entirely on this...

Friday, June 14, 13

Chefs on a Plane

What if...

...even while aboard this?




upload+



list+





upload(


of this...

...to run entirely on this...

Friday, June 14, 13

Source'Code'

Compiler'

Ar/fact'

Test'

Developing for Chef: Rapid Iteration

• Less time waiting around for cookbook deploys and Chef runs

• More frequent testing

• Better code

• Business needs met more quickly

More Fun!

Less Boring!

X

Friday, June 14, 13

Code Can (Help)Friday, June 14, 13

The Toolchain

Friday, June 14, 13

The Toolchain

• You’re becoming a developer!• Reasonably powerful computer• Good editor

Friday, June 14, 13

Have a Good Computer• Virtualization is used to run

acceptance tests• Running on a real virtualized

“node”• Lots of memory (4GB+, 8GB

recommended)• Fast disk (SSD)• Good processor (Intel i5+)• Modern OS

XFriday, June 14, 13

Tools• Chef (obviously)• Virtualization provider• Virtualization automation

(Vagrant)• Cookbook dependency

manager (Berkshelf, Librarian)

• Unit and acceptance testing frameworks

Friday, June 14, 13

Software Testing Terminology and Chef

Friday, June 14, 13

Different Types of Testing

• There are formal Software Engineering definitions for testing

Friday, June 14, 13


• There are formal Software Engineering definitions for testing• Unit test

Friday, June 14, 13


• There are formal Software Engineering definitions for testing• Unit test• Integration test

Friday, June 14, 13


• There are formal Software Engineering definitions for testing• Unit test• Integration test• Acceptance test

Friday, June 14, 13


• There are formal Software Engineering definitions for testing• Unit test• Integration test• Acceptance test

• An easy explanation for SAs is ...

Friday, June 14, 13

• Unit Test: Signal Input

What and When To Test

Flickr user: Rain Rabbit

Friday, June 14, 13


• Did I send Chef the correct command?



Friday, June 14, 13



• Signal Processing



Friday, June 14, 13




• Did Chef interpret my command correctly?



Friday, June 14, 13





• Acceptance Test: Signal Output



Friday, June 14, 13





• Acceptance Test: Signal Output

• Did my expressed intent, executed by Chef, achieve the desired result?



Friday, June 14, 13

Chef Testing Tools

Friday, June 14, 13

Chef Testing Tools

• Signal Input• ChefSpec

Friday, June 14, 13

Chef Testing Tools


• Signal out• Chef Minitests & Minitest Handler

Friday, June 14, 13

Chef Testing Tools


• Signal out• Chef Minitests & Minitest Handler

• Signal processing• Who cares; that’s what Opscode’s own tests for

Chef are for!

Friday, June 14, 13

General Philosophy for Chef Testing

Friday, June 14, 13


• Unit tests• ChefSpec assertion for every resource of interest• Each recipe should get its own test file

Friday, June 14, 13



• Acceptance tests• Anything that you couldn’t test in a unit test

Friday, June 14, 13



• Acceptance tests• Anything that you couldn’t test in a unit test

• Don’t repeat yourself!

Friday, June 14, 13

Unit Testing: ChefSpec

Friday, June 14, 13

ChefSpec Overview

• Built on top of RSpec• Converge a Chef run in memory• Overrides all providers to take no action• Make assertions about the in-memory Chef run• Mock Ohai data (automatic attributes) with Fauxhai

Friday, June 14, 13

Example Recipe Part Oneinclude_recipe “java”

user node['sauceproxy']['server']['user'] do comment "SauceLabs Proxy User" system true action :createend

directory node['sauceproxy']['server']['install_dir'] do owner node['sauceproxy']['server']['user'] mode 00755 action :createend

# Can't assume we have unzippackage "unzip" do action :installend

execute "unzip-saucelabs-proxy" do cwd node['sauceproxy']['server']['install_dir'] command "unzip -o #{Chef::Config[:file_cache_path]}/#{node['sauceproxy']['server']['zipfile']}" action :nothing notifies :restart, "service[sauceproxy]"end

https://github.com/juliandunn/sauceproxy/blob/master/recipes/server.rb

Friday, June 14, 13



Example Recipe Part Tworemote_file "#{Chef::Config[:file_cache_path]}/#{node['sauceproxy']['server']['zipfile']}" do

source "#{node['sauceproxy']['server']['download_url']}/#{node['sauceproxy']['server']['zipfile']}" action :create_if_missing notifies :run, "execute[unzip-saucelabs-proxy]", :immediatelyend

template "/etc/init.d/sauceproxy" do source "sauceproxy.init.erb" mode 00755 owner "root" group "root" action :createend

template "/etc/sysconfig/sauceproxy" do source "sauceproxy.sysconfig.erb" mode 00644 owner "root" group "root" action :createend

service "sauceproxy" do supports :restart => true action [:enable, :start]end

Friday, June 14, 13

Example ChefSpec Test: Setupdescribe 'sauceproxy::server' do

let (:chef_run) do runner = ChefSpec::ChefRunner.new( platform: 'centos', version: '6.3' ) runner.node.set['sauceproxy']['server']['user'] = 'fake' runner.node.set['sauceproxy']['server']['install_dir'] = '/tmp/fake' runner.node.set['sauceproxy']['server']['version'] = '3.14159' runner.converge('sauceproxy::server') end

Friday, June 14, 13

Example ChefSpec Test: Expectations it 'should create a sauceproxy directory with the right ownership' do expect(chef_run).to create_directory('/tmp/fake') expect(chef_run.directory('/tmp/fake')).to be_owned_by('fake') end

it 'should start a service called sauceproxy' do expect(chef_run).to start_service('sauceproxy') expect(chef_run).to set_service_to_start_on_boot('sauceproxy') endend

Friday, June 14, 13

ChefSpec In Action

Friday, June 14, 13

ChefSpec: Expectations

• Many expectations (assertions) possible• https://github.com/acrmp/chefspec#making-

assertions• Don’t (just) restate static resources

• Mock out data• Static resource assertions: good for regressions

• Whoops, I deleted the service resource

Friday, June 14, 13

https://github.com/acrmp/chefspec#

https://github.com/acrmp/chefspec#

Acceptance Testing: MiniTest

Friday, June 14, 13

Chef Client Run

Friday, June 14, 13

Chef Client Run

Friday, June 14, 13

Acceptance Tests as Report Handler

• Acceptance testing as a Chef Report Handler• Many tools (serverspec, bats, minitest)• I’ll demo Chef-MiniTest-Handler; most widely used

Friday, June 14, 13

MiniTest Handler Cookbook

• http://community.opscode.com/cookbooks/minitest-handler• Installs Minitest Gems• Installs Chef Minitest Gems• Installs the Chef-Minitest-Handler Notification

Handler for Chef-Client• Places test files from cookbooks on the target node

as part of a Chef-client run

Friday, June 14, 13

http://community.opscode.com/cookbooks/minitest-handler




MiniTest Files

• Each recipe gets an individual test file.• recipes/server.rb• files/default/test/server_test.rb

• Tests for each recipe are automatically loaded by the handler

Friday, June 14, 13

Example MiniTestrequire 'minitest/spec'

describe_recipe 'sauceproxy::server' do

it 'runs as a daemon' do service('sauceproxy').must_be_running end

end

http://tinyurl.com/minitest-examplesfor many more example tests

Friday, June 14, 13

http://tinyurl.com/minitest-examples




Make this go on a

with virtualization!Friday, June 14, 13

Using Vagrant & Berkshelf to Iterate

• Vagrant is virtualization middleware• Driven from Vagrantfile• Defines VM images, customization parameters,

provisioners (Chef in our case)• Berkshelf is a cookbook dependency manager

• Get dependent cookbooks from community API• Feed them to Vagrant Chef provisioner

Friday, June 14, 13

Example VagrantfileVagrant.configure("2") do |config| config.vm.hostname = "sauceproxy-berkshelf" config.vm.box = "opscode-centos-6.4" config.vm.box_url = "https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-6.4_chef-11.4.4.box" config.ssh.max_tries = 40 config.ssh.timeout = 120 config.vm.provision :chef_solo do |chef| chef.json = {} chef.run_list = [ "recipe[minitest-handler]", "recipe[sauceproxy::server]" ] endend

Friday, June 14, 13

https://dl.dropbox.com/u/47541301/vagrantboxes/centos-6.3.box




% vagrant up

Bringing Up Vagrant

borkbork:~/Dropbox/devel/github/juliandunn/sauceproxy (master)$ vagrant upBringing machine 'default' up with 'virtualbox' provider...[default] Importing base box 'centos-6.4'...[default] Matching MAC address for NAT networking...[default] Setting the name of the VM...[default] Clearing any previously set forwarded ports...[Berkshelf] Updating Vagrant's berkshelf: '/Users/juliandunn/.berkshelf/vagrant/berkshelf-20130611-20810-rt7k01'[Berkshelf] Using sauceproxy (0.1.8) at path: '/Users/juliandunn/Dropbox/devel/github/juliandunn/sauceproxy'[Berkshelf] Using minitest-handler (0.2.1)[Berkshelf] Using java (1.11.4)[Berkshelf] Using windows (1.8.10)[Berkshelf] Using chef_handler (1.1.4)[default] Creating shared folders metadata...[default] Clearing any previously set network interfaces...[default] Preparing network interfaces based on configuration...[default] Forwarding ports...[default] -- 22 => 2222 (adapter 1)[default] Booting VM...[default] Waiting for VM to boot. This can take a few minutes.

Friday, June 14, 13

% vagrant up

Bringing Up Vagrant

borkbork:~/Dropbox/devel/github/juliandunn/sauceproxy (master)$ vagrant upBringing machine 'default' up with 'virtualbox' provider...[default] Importing base box 'centos-6.4'...[default] Matching MAC address for NAT networking...[default] Setting the name of the VM...[default] Clearing any previously set forwarded ports...[Berkshelf] Updating Vagrant's berkshelf: '/Users/juliandunn/.berkshelf/vagrant/berkshelf-20130611-20810-rt7k01'[Berkshelf] Using sauceproxy (0.1.8) at path: '/Users/juliandunn/Dropbox/devel/github/juliandunn/sauceproxy'[Berkshelf] Using minitest-handler (0.2.1)[Berkshelf] Using java (1.11.4)[Berkshelf] Using windows (1.8.10)[Berkshelf] Using chef_handler (1.1.4)[default] Creating shared folders metadata...[default] Clearing any previously set network interfaces...[default] Preparing network interfaces based on configuration...[default] Forwarding ports...[default] -- 22 => 2222 (adapter 1)[default] Booting VM...[default] Waiting for VM to boot. This can take a few minutes.

Friday, June 14, 13

Vagrant Run Continued[default] Waiting for VM to boot. This can take a few minutes.[default] VM booted and ready for use!GuestAdditions 4.2.12 running --- OK.[default] Setting hostname...[default] Configuring and enabling network interfaces...[default] Mounting shared folders...[default] -- /vagrant[default] -- /tmp/vagrant-chef-1/chef-solo-1/cookbooks[default] Running provisioner: chef_solo...Generating chef JSON and uploading...Running chef-solo...[2013-06-12T02:52:26+00:00] INFO: *** Chef 11.4.4 ***[2013-06-12T02:52:26+00:00] INFO: Setting the run_list to ["recipe[minitest-handler]", "recipe[sauceproxy::server]"] from JSON[2013-06-12T02:52:26+00:00] INFO: Run List is [recipe[minitest-handler], recipe[sauceproxy::server]][2013-06-12T02:52:26+00:00] INFO: Run List expands to [minitest-handler, sauceproxy::server][2013-06-12T02:52:26+00:00] INFO: Starting Chef Run for sauceproxy-berkshelf

Friday, June 14, 13

Vagrant Run Continued[2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds[2013-06-12T02:55:22+00:00] INFO: Running report handlersRun options: -v --seed 31883

# Running tests:

recipe::java::openjdk#test_0001_installs the correct version of the jdk =0.11 s = .recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =0.04 s = .recipe::sauceproxy::server#test_0001_runs as a daemon =0.09 s = .

Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.

3 tests, 3 assertions, 0 failures, 0 errors, 0 skips[2013-06-12T02:55:22+00:00] INFO: Report handlers complete

Friday, June 14, 13

Vagrant Run Continued[2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds[2013-06-12T02:55:22+00:00] INFO: Running report handlersRun options: -v --seed 31883

# Running tests:




Friday, June 14, 13

Minitests from Java cookbook ran too![2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds[2013-06-12T02:55:22+00:00] INFO: Running report handlersRun options: -v --seed 31883

# Running tests:




Friday, June 14, 13

Minitests from Java cookbook ran too![2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds[2013-06-12T02:55:22+00:00] INFO: Running report handlersRun options: -v --seed 31883

# Running tests:




Friday, June 14, 13

New Test/Dev CycleWrite Unit Tests

Worked?

Run ChefSpec

No

Write Recipe Code

Write Acceptance Tests

Yes

Worked?

Run Vagrant + Minitest Handler

No

vagrant destroy

Yes Commit/Tag Code, etc.

Friday, June 14, 13

CI Pipeline Integration

Friday, June 14, 13

Continuous Integration Pipeline

• Run unit tests in your pipeline • Run acceptance tests in your

pipeline• Drive cookbook uploads as the

output!

Friday, June 14, 13

Travis-CI Example for SauceProxylanguage: rubygemfile: - gemfiles/travis.gemfilervm: - "1.9.2" - "1.9.3"script: bundle exec rake test:syntax test:lint test:specnotifications: email: - [email protected]

Friday, June 14, 13



Rakefile Example

• Too long to post here• Rake tasks for Foodcritic

and ChefSpec• https://github.com/

juliandunn/sauceproxy/blob/master/Rakefile

Friday, June 14, 13

https://github.com/juliandunn/sauceproxy/blob/master/Rakefile






Travis-CI Example Dashboard

Friday, June 14, 13

^#$%(#$ I broke the build

Friday, June 14, 13

Test Kitchen (alpha)

• Run acceptance test suite on multiple OSes

• Different fixtures for each, if desired• Different drivers (vagrant, ec2, lxc,

etc.)• Hook up to Jenkins or other CI

system if desired

Friday, June 14, 13

Test Kitchen Config File---driver_plugin: vagrantplatforms:- name: centos-6.4 driver_config: box: opscode-centos-6.4 box_url: https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-6.4_provisionerless.box require_chef_omnibus: true- name: centos-5.9 driver_config: box: opscode-centos-5.9 box_url: https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-5.9_provisionerless.box require_chef_omnibus: true

suites:- name: default run_list: ["recipe[minitest-handler]", "recipe[sauceproxy::server]"] attributes: {}

Friday, June 14, 13

https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-6.4_provisionerless.box




Test Kitchen Demo

Friday, June 14, 13

Resources

Friday, June 14, 13

Resources• ChefSpec: https://github.com/acrmp/chefspec• Chef Minitest Handler: https://github.com/calavera/

minitest-chef-handler • Berkshelf: http://berkshelf.com/• Vagrant: http://vagrantup.com/• Test Kitchen: https://github.com/opscode/test-kitchen• Bento: https://github.com/opscode/bento• SauceProxy Sample Cookbook: https://github.com/

juliandunn/sauceproxyFriday, June 14, 13

https://github.com/acrmp/chefspec

https://github.com/acrmp/chefspec

http://download.vagrantup.com




http://vagrantup.com

http://vagrantup.com

https://github.com/opscode/test-kitchen

https://github.com/opscode/test-kitchen

https://github.com/juliandunn/sauceproxy




More on Testing

• Chef-NYC: The Hows and Whys of Cookbook Testing

• Seth Vargo (Author of ChefSpec, Fauxhai, many others)

• http://www.meetup.com/Chef-NYC/events/122219772/

• June 25th in Manhattan

Friday, June 14, 13

http://www.meetup.com/Chef-NYC/events/122219772/




Chef Fundamentals Training

• Boston, July 11-12 (CompuWorks, 263 Summer St.)

• eventbrite.com/event/6652057483

• Use code BOSTON-MEETUP to save 25%!

Friday, June 14, 13

http://www.eventbrite.com/event/6652057483#








Questions and PrizesAlso, we’re hiring! Automate all the things for fun and profit!http://www.opscode.com/blog/careers/(and/or see me after)

Friday, June 14, 13

http://www.opscode.com/blog/careers/






Technology

Chef Cookbook Testing and Continuous Integration