Test Driven Infrastructure with Docker, Test Kitchen and Serverspec Yury Tsarev

Agenda

▸ Goal▸ Test kitchen▸ Implementation at GoodData▸ Create test-driven infrastructure change▸ Wrap-Up

Goal

▸ Infrastructure code should be treated as any other code▸ Apply TDD for puppet▸ Grow regression test suite

Test Kitchen

▸ http://kitchen.ci/ ▸ Test orchestrator▸ Originated in Chef community▸ Very pluggable on all levels▸ "Your infrastructure deserves tests too."▸ Book - Test-Driven Infrastructure with Chef

Test Kitchen - High Level Process I

1. Create VM/Container2. Run configuration management code there 3. Run the test suite

Test Kitchen - High Level Process II

Test Kitchen

Drivercreate

Provisionerconverge

Verifierverify

Instance Under Test

Configuration File - .kitchen.yml

▸ Driver: what type of VM/containerization/cloud to use▸ Amazon EC2, Blue Box, CloudStack, Digital Ocean, Rackspace, OpenStack,

Vagrant, Docker, LXC containers

▸ Provisioner: which configuration management tool to apply▸ Chef, Puppet, Ansible, SaltStack

▸ Verifier: test automation type to verify with▸ Bats, shUnit2, RSpec, Serverspec

▸ Transport: mechanism to upload files into instance under test

▸ Platform: os/environment to run tests on▸ Suite: test suite to execute

Implementation at GoodData

▸ Docker driver▸ Puppet provisioner▸ SFTP Transport▸ Serverspec verifier

Infrastructure Test Pipeline in GD

Puppet Catalog

CompilationTest Kitchen

Docker Container

Puppet Serverspec

VM Instance in Dev/QA/Staging Env

Puppet Serverspec

Shellmocking

Docker Container

Puppet Serverspec

Shellmocking

Docker Driver

▸ https://github.com/portertech/kitchen-docker▸ kitchen driver to work with docker containers as machines

under testdriver:

name: docker

image: docker-registry.example.com:80/gdc:R23

platform: rhel

use_sudo: false

provision_command:

yum clean all && yum makecache

Puppet Provisioner

▸ https://github.com/neillturner/kitchen-puppet▸ Uploads puppet code into instance▸ Runs puppet there - converge▸ Provides facts customization facility

Puppet Provisioner - Example

provisioner:

name: puppet_apply

require_chef_for_busser: false

modules_path: puppet/modules

manifests_path: puppet/manifests

hiera_data_path: puppet/hieradata

facterlib: /etc/puppet/facter

install_custom_facts: true

custom_facts:

docker: 1

ec2data_freeipa_otp: test

ec2data_nopuppet_cron: 1

ec2_public_ipv4: 127.0.0.1

# our custom fact for use in serverspec call

serverspec_check_type: role

Transport

▸ https://github.com/coderanger/kitchen-sync▸ Replaced default scp with sftp transport▸ Reduced 1.5m to 5 sec for puppet and test suite upload

transport:

name: sftp

remote_ruby_path: 'ruby193-ruby'

username: kitchen

Serverspec Test Suite

▸ http://serverspec.org/ - RSpec based framework▸ http://serverspec.org/resource_types.html ▸ Independent of kitchen - can be used standalone▸ We keep test suite together with puppet code under

spec directory to have ability to create consistent PRs

Serverspec DSL Examples

describe file('/var/log/httpd') do

it { should be_directory }

end

describe command('apachectl -M') do

its(:stdout) { should contain

('proxy_module')}

end

describe default_gateway do

its(:ipaddress) { should eq '192.168.10.1'

}

its(:interface) { should eq 'br0'

}

end

describe cgroup('group1') do

its('cpuset.cpus') { should eq 1 }

end

describe docker_container('focused_curie')

do

its(:HostConfig_NetworkMode) { should eq

'bridge' }

its(:Path) { should eq '/bin/sh' }

end

describe port(53) do

it { should be_listening.with('udp') }

end

Serverspec Verifier: Busser or Shell

▸ Simple

https://github.com/test-kitchen/busser-serverspec

▸ Advanced

https://github.com/vincentbernat/serverspec-example

Serverspec Verifier in GD

▸ Based on shell verifier and serverspec test suite▸ Runs a serverspec test suite against the configured

(converged) instance▸ Reports in shell and junit - jenkins ready

verifier:

name: shell

remote_exec: true

command: |

sudo -s <<SERVERSPEC

export SERVERSPEC_ENV=$EC2DATA_ENVIRONMENT

export SERVERSPEC_BACKEND=exec

serverspec junit=true tag=~skip_in_kitchen check:role:$EC2DATA_TYPE

SERVERSPEC

Shellmocking

▸ To bypass external dependencies and docker specific limitations

▸ Simple ruby script▸ Wraps package manager invocation▸ Mock is defined in simple yaml format:

package:

/path/to/executable: contents

Shellmock ExampleConfiguration Resulting mock

ipa-client:

/usr/sbin/ipa-client-install: |

echo 'server = freeipa.example.

com' > /etc/ipa/default.conf

echo 'fake' > /etc/krb5.keytab

$ cat /usr/sbin/ipa-client-install

echo 'server = freeipa01.intgdc.

com' > /etc/ipa/default.conf

echo 'fake' > /etc/krb5.keytab

sssd:

/usr/sbin/sssd:

$ cat /usr/sbin/sssd

#!/bin/bash

echo I am a fake /usr/sbin/sssd

Linux Kernel Capabilities

▸ Sometimes it is necessary to enable kernel capabilities to make things working in container

driver:

cap_add:

- SYS_RESOURCE

▸ In the above example we are allowing container to set NPROC resource limit

▸ See `man capabilities` for an actual list for you kernel version

Defining Puppet Type in Kitchen

▸ Declare in platforms section of .kitchen.ymlplatforms:

- name: zuul

provisioner:

custom_facts:

ec2data_type: zuul

▸ kitchen converge▸ shellmock▸ repeat

Create Test-Driven Infrastructure Change

▸ Write serverspec expectation for new code▸ kitchen verify <type>▸ Observe related test is red▸ Write puppet code▸ kitchen converge <type>▸ kitchen verify <type>▸ Observe related test is green▸ Commit the changes and create PR to puppet repo▸ DEMO

WTF? Why should I write the same thing two times both in puppet and serverspec?

▸ It pays off at scale▸ It creates Red-Green-Refactor TDD flow

▸ Testable code▸ Regression test suite

▸ Ideally you are not only repeating matching puppet declarations in serverspec but testing the outcomeunder_kerberos_auth = ['', 'status', 'status.json', 'status/payload', 'images/grey.png']

under_kerberos_auth.each do |path|

describe command("curl -k -X POST http://127.0.0.1/#{path}") do

its(:stdout) { should match(/301/) }

end

describe command("curl -k -X POST https://127.0.0.1/#{path}") do

its(:stdout) { should match(/401/) }

its(:stdout) { should match(/Authorization Required/) }

end

end

Wrap-Up: Benefits

▸ Scratch environment▸ Test in isolation▸ Easy to test permutations▸ Resource efficiency▸ Test-first/test-driven approach for infrastructure code▸ Fast feedback - even before a commit▸ Naturally growing regression test suite▸ Easily pluggable into CI/CD pipeline

Wrap-Up: OSS Side of Things

▸ Multiple open source projects combined▸ We contributed a lot into related kitchen projects and

serverspec▸ Nominated as kitchen-puppet core contributor

Wrap-Up: What is Next ?

▸ Even with containers it is still resource costly operation▸ We use automation based on puppet-catalog-diff to

determine which puppet types are affected by Pull Request▸ Then only affected types are tested by kitchen

Wrap-Up: A Note on Agnostic Approach

▸ As the kitchen driver is agnostic to virtualization solution/cloud provider

▸ As the kitchen provisioner is agnostic to configuration management solution

▸ As the serverspec is agnostic to configuration type at all▸ We are capable to make brave movements in future!

THANKS! Questions?

▸ http://kitchen.ci/ ▸ https://github.com/portertech/kitchen-docker ▸ https://github.com/neillturner/kitchen-puppet ▸ http://serverspec.org/