PuppetConf 2016: The Future of Testing Puppet Code – Gareth Rushgrove, Puppet

(without introducing more risk)

The Future of Testing Puppet Code

PuppetGareth Rushgrove

Who, What, Why, How


@garethr


Gareth Rushgrove

(without introducing more risk)This TalkWhat we’ll cover

- Why test Puppet anyway- Existing Puppet testing tools- Different users and usecases- Future challenges

Gareth Rushgrove

Mainly a talk about problems,rather than solutions

Gareth Rushgrove

Not a talk about how to testPuppet code. Lot of existing content on that subject

Gareth Rushgrove


Gareth Rushgrove

PuppetConf 2014


Gareth Rushgrove

Survey of 300 Puppet users


Gareth Rushgrove

(without introducing more risk)Why Test Puppet

And a little history

Gareth Rushgrove

- You have lots of Puppet code- You’re targeting a wide range of OS’s- The Puppet language has more features

Gareth Rushgrove


Gareth Rushgrove

Lots of codeBut how much is lots

Note that this just represents public code on GitHub. Lots of private Puppet code out there too

Gareth Rushgrove


Gareth Rushgrove

How many .pp files on GitHub?

SELECT COUNT(*) FROM [puppet.puppet_files]

-- 489725


Gareth Rushgrove

How many lines of Puppet code?

SELECT COUNT(line) total_lines FROM ( SELECT SPLIT(content, '\n') AS line FROM [puppet.puppet_content] )

-- 6697099


Gareth Rushgrove

How big are .pp files?

SELECT QUANTILES(total_lines, 10) AS q FROM ( SELECT COUNT(line) total_lines FROM ( SELECT SPLIT(content, '\n') AS line, id FROM [puppet.puppet_content] ) GROUP BY id )

50% of Puppet files are 20 lines or less. 90% of Puppet files are 100 lines or less

Gareth Rushgrove

The largest Puppet file public on GitHub has 25,238 lines

Gareth Rushgrove


Gareth Rushgrove

Mixed environmentsSo many platforms

Gareth Rushgrove

A third of respondents run more than one OS on there development machine

Gareth Rushgrove

Gareth Rushgrove

60% of respondents run more than one OS in production

Gareth Rushgrove


Gareth Rushgrove

The Puppet language

Types, lambdas, iterators, functions

Gareth Rushgrove

(without introducing more risk)Existing Tools

Providing some context

Gareth Rushgrove

(without introducing more risk)Core tools

Common starting points

Gareth Rushgrove


Gareth Rushgrove

puppet-lint


Gareth Rushgrove

Style conformance

$ puppet-lint /etc/puppet/modules foo/manifests/bar.pp - ERROR: trailing whitespace found on line 1 apache/manifests/server.pp - WARNING: variable not enclosed in {} on line 56 ...


Gareth Rushgrove

rspec-puppet


Gareth Rushgrove

Unit tests for Puppet

it { is_expected.to compile } it { is_expected.to contain_class('db') } it do is_expected.to contain_package('mysql-server') .with_ensure('present') end it { is_expected.to have_exec_resource_count(0) }


Gareth Rushgrove

puppet-syntax


Gareth Rushgrove

Popular Puppet gemsSELECT package, COUNT(*) count FROM ( SELECT REGEXP_EXTRACT(line, r'gem [\'|"](.*)[\'|"]') package, id FROM ( SELECT SPLIT(content, '\n') line, id FROM [puppet.gemfile_contents] WHERE content CONTAINS 'gem' ) GROUP BY package, id ) GROUP BY 1 ORDER BY count DESC LIMIT 30;


Gareth Rushgrove

Popular gem results

puppetlabs_spec_helper 1157 puppet 1142 rake 1135 puppet-lint 768 beaker-rspec 610 rspec-puppet 576 metadata-json-lint 566 beaker 440 puppet-blacksmith 425 serverspec 359


Gareth Rushgrove

puppetlabs_spec_helper


Gareth Rushgrove

Advanced toolsNext steps


Gareth Rushgrove

beaker-rspec


Gareth Rushgrove

Serverspec


Gareth Rushgrove

rubocop


Gareth Rushgrove

Other toolsAlternative approaches


Gareth Rushgrove

puppet-catalog-test


Gareth Rushgrove

Test catalogue compilation

$ puppet-catalog-test -m modules -M site.pp [INFO] Using puppet 3.0.2 [PASSED] foo (compile time: 0.168182 seconds) [PASSED] default (compile time: 0.003451 seconds)

---------------------------------------- Compiled 2 catalogs in 0.1717 seconds (avg: 0.0858 seconds)


Gareth Rushgrove

Test Kitchen

(without introducing more risk)Different Users

And different usecases

Gareth Rushgrove

Half of respondents regularly writetests for Puppet code

Gareth Rushgrove

A quarter of respondents never write tests for Puppet code

Gareth Rushgrove

The problem with a tool-centricworld view

Gareth Rushgrove

Understanding user needs meansstarting with users

Gareth Rushgrove

Gareth Rushgrove

- The Vox Pupuli folks- Lots of Puppet employees- You in-house Puppet expert- Custom functions

The module authors

The regular Puppet users

Gareth Rushgrove

- Roles and profiles- Using Puppet weekly or more- PuppetDB, Puppet Server- Custom facts

The occasional Puppet user

Gareth Rushgrove

- Using Puppet as needed- Basic Puppet code- Hieradata

Today different types of Puppet users have a very different experience

Gareth Rushgrove

Gareth Rushgrove

For those with no Ruby experience the average was 2 out of 5

Gareth Rushgrove

For those with a little Ruby experience the average was still only 2.5 out of 5

Gareth Rushgrove

Of the people who love the testing experience 33% are self-described power users of both Puppet and Ruby

Gareth Rushgrove

Gareth Rushgrove

But more than 50% of respondents have little or no Ruby experience

Gareth Rushgrove

(without introducing more risk)Challenges

What we need for the future


Gareth Rushgrove

Improved getting started experience

Helping a new generation of users

- Getting started testing documentation- Single entry point for users- Opinionated workflow

Gareth Rushgrove


Gareth Rushgrove

Adopting acceptance testing

Still too hard

27 of the 67 Serverspec users are not using either Beaker or Test Kitchen.

Gareth Rushgrove

Only 5 of the Beaker (without rspec) users are not also using beaker-rspec

Gareth Rushgrove

3 of the Test Kitchen users are also using some flavour of Beaker

Gareth Rushgrove

So only 26% to 35% of respondents are using an acceptance testing tool with there Puppet code

Gareth Rushgrove


Gareth Rushgrove

Experience mattersConsistence interfaces


Gareth Rushgrove

Inconsistencies

$ puppet-lint vs rake lint vs rake validate $ rake syntax vs puppet parser validate $ rspec vs rake spec $ BEAKER_destroy=no rake beaker vs beaker

We have a language style guide. What about other Puppet user interfaces?

Gareth Rushgrove


Gareth Rushgrove

Ruby heritageRuby as a development environment

60% of respondents described themselves as Puppet power users

Gareth Rushgrove

Only 12% of respondents described themselves as Ruby power users. A third of those work for Puppet!

Gareth Rushgrove

- More help with using Ruby for testing?- Support for testing in other languages?- Native Puppet support for testing?- Containerised testing abstraction?

Gareth Rushgrove


Gareth Rushgrove

jordan/spec


Gareth Rushgrove

Test Puppet with Puppet

file { '/tmp/test': ensure => present, content => 'The file content', }

assertion { 'that the file has the correct contents': subject => File['/tmp/test'], attribute => 'content', expectation => 'The file content', }

(without introducing more risk)Conclusions

Where to go from here

Thanks to the community we have a collection of powerful testing tools in the Puppet ecosystem

Gareth Rushgrove

But we don’t have consistent user interfaces or the opinionated workflows useful for beginners

Gareth Rushgrove

Testing shouldn’t just be apower user feature

Gareth Rushgrove

As a community we should talkabout design more often

Gareth Rushgrove


Gareth Rushgrove

https://goo.gl/forms/nchOp5hYdLLgmJ8z1

Gareth Rushgrove

(without introducing more risk)Questions?And thanks for listening

Technology

PuppetConf 2016: The Future of Testing Puppet Code – Gareth Rushgrove, Puppet