Configuration Management with Puppet

Configuration Management with Puppet

Rachel Andrew, Future Insights Live! 2015

Photo credit: https://www.flickr.com/photos/andreakirkby/5450450019

https://www.flickr.com/photos/andreakirkby/5450450019

@rachelandrew | Read more at http://rachelandrew.co.uk/presentations/puppet-developers

Rachel Andrew

http://rachelandrew.co.uk

@rachelandrew

http://grabaperch.com

http://rachelandrew.co.uk/presentations/puppet-developers

http://rachelandrew.co.uk

http://grabaperch.com


Many issues coming into support stem from poor development and deployment processes.



The Big Problems

• Developing directly on live sites or in subfolders of live sites

• Developing in subfolders locally

• Setting up local development environments that are so different to the eventual live server that there is no confidence when going live

• Working in teams where everyone has a slightly different setup



Improving workflow is the best way to give yourself more hours in the day.



Today we’re going to take a look at

• A really simple way to get started with Vagrant and Puppet for those of you who haven’t used these tools before.

• Vagrant and Puppet fundamentals, how to start writing your own manifests.

• How you can take this knowledge into production, even if you manage just one or two servers.



First Steps



We need …

• to develop multiple websites on our own computer.

• to know that our live server and local server support the same things.

• to deploy our site and have confidence that what is on the live server is identical to our local version.

• to have everyone who works on a site using the same development environment so we aren’t creating problems for each other.



Q. How do you develop sites locally that require a web server?

A. MAMP, WAMP, XAMPP or similar

63%



In an ideal world your local development environment is identical to the live server.



http://php.net/manual/en/function.strftime.php


http://php.net/manual/en/function.strftime.php


PHP Versions can be different. PHP modules may not be available on the live server.



Virtual Machines



virtualbox.org


http://virtualbox.org


vagrantup.com


http://vagrantup.com


https://puphpet.com


https://puphpet.com




http://rachelandrew.co.uk/presentations/deploy/puphpet


http://rachelandrew.co.uk/presentations/deploy/puphpet


Upload a PHP file with this function to find out what is available on your live server.

<?php phpinfo();?>



Things to check on your live server

• PHP Version

• Installed modules such as gd for image processing

• post_max_size and upload_max_filesize determine the size of files that can be uploaded

• max_input_vars is the number of form fields allowed in a post



Shared Folders











Basic commands for Vagrant.

// start the VM > vagrant up

// shut down the VM > vagrant halt

//destroy the VM > vagrant destroy

//ssh access > vagrant ssh



http://vagrantmanager.com/


http://vagrantmanager.com/


Share your package with anyone else working on the site. The whole team can then have the

exact same development setup.



Using PuPHPet should get you up and running in a few hours. This will pay dividends in time

saved in the long run.



Getting Under the Hood



https://github.com/PerchCMS/perch-vagrant






In the Vagrantfile we tell Vagrant we will be provisioning with Puppet.

# Enable the Puppet provisioner, point it to our files config.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/manifests" puppet.manifest_file = "site.pp" puppet.module_path = "puppet/modules" puppet.hiera_config_path = "puppet/hiera.yaml" end



What is Puppet?

• A Configuration Management solution

• Allows you to define in code the state of a server including

• Packages that should be installed

• Services that should be running

• Files and folders

• Permissions

• Cron jobs



You could set up a VM and then install everything by hand using the package manager for your distribution.

> sudo apt-get install apache2 > sudo apt-get install mysql-server > sudo apt-get install php5 php5-mysql





Puppet Terminology

• Manifest: a file that contains Puppet code

• Resource: a thing that needs configuring, Apache is a resource, and so is a virtual-host. Resources have types - for example file, package, cron.

• Module: a collection of manifests, templates and other files organised around a particular purpose.



Inside the Apache module is a manifests folder. This contains the manifests:

- init.pp - vhost.pp puppet

modules apache manifests - init.pp - vhost.pp templates - vhost-default-conf.erb



All modules need an init.pp manifest. It is used when the module is included.

In the apache init.pp

- install the apache package - make sure apache is running - install the rewrite module

class apache {

package { "apache2": ensure => present, }

service { "apache2": ensure => running, require => Package["apache2"], }

file { "/etc/apache2/mods-enabled/rewrite.load": ensure => link, target => "/etc/apache2/mods-available/rewrite.load", require => Package["apache2"] }

}


http://garylarizza.com/blog/2014/10/19/on-dependencies-and-order/

On dependencies and order - why Puppet doesn’t care about execution order (until it does).


http://garylarizza.com/blog/2014/10/19/on-dependencies-and-order/



The manifest file vhost.pp sets up a VirtualHost by creating a file in sites-available and symlinking it into sites-enabled.

We notify the apache2 service, which will then reload to pick up the new config.

define apache::vhost( $vhost_docroot = false, $vhost_name = false, $vhost_options =['Indexes','FollowSymLinks','MultiViews'],

) { file {"/etc/apache2/sites-available/${vhost_name}": content => template("apache/vhost-default.conf.erb"), owner => 'root', group => 'root', mode => '755', require => Package['apache2'], notify => Service['apache2'];

"/etc/apache2/sites-enabled/${vhost_name}": ensure => link, target => "/etc/apache2/sites-available/${vhost_name}", notify => Service['apache2']; } }



The template file for a VirtualHost includes variables which will be replaced out by the details for each host.

# ************************************ # Default template for vhosts # Managed by Puppet # ************************************

<VirtualHost *:80> ServerName <%= @vhost_name %> DocumentRoot <%= @vhost_docroot %> <Directory <%= @vhost_docroot %>> Options <%= @vhost_options %> AllowOverride All Order allow,deny allow from all </Directory> ErrorLog /var/log/apache2/<%= @vhost_name %>_error.log LogLevel warn CustomLog /var/log/apache2/<%= @vhost_name %>_access.log combined ServerSignature Off </VirtualHost>







The site.pp file is special and kicks off the whole process.

In my case it includes the modules we want to run.

stage { 'setup': before => Stage['main'] }

class { 'base': stage => 'setup' }

include base, apache, mysql, php, bootstrap







The Hiera config file defines a YAML backend and gives the location of the configuration data.

In my project that is in the manifest directory, in a folder named hiera.

--- :backends: yaml :yaml: :datadir: "%{settings::manifestdir}/hiera" :hierarchy: - "%{::clientcert}" - "%{::environment}" - config :logger: console





In my YAML file I have added a setting for mysql_root_password.

File: manifests/hiera/config.yaml

mysql_root_password: 'vagrant'



I set the parameter $root_pw with the value using a hiera() function.

I can then use $root_pw within the manifests.

File: modules/mysql/manifests/init.pp

class mysql( $root_pw = hiera('mysql_root_password'), ) { package { "mysql-server": ensure => present, }

service { "mysql": enable => true, ensure => running, require => Package["mysql-server"], }

exec { "set-mysql-password": unless => "/usr/bin/mysqladmin -uroot -p$root_pw status", command => "/usr/bin/mysqladmin -uroot password $root_pw", require => Service["mysql"], } }



In my YAML file I have a list of PHP Modules.

File: manifests/hiera/config.yaml

php_modules: - "php5" - "php5-cli" - "php5-mysql" - "php5-gd" - "php5-imagick" - "php5-curl" - "libapache2-mod-php5"



I get the php_modules with the hiera function and then pass the list to the package resource type.

File: modules/php/manifests/init.pp

class php( $packages = hiera('php_modules'), $php_upload_max_filesize = hiera('php_upload_max_filesize'), $php_max_file_uploads = hiera('php_max_file_uploads'), $php_memory_limit = hiera('php_memory_limit'), $php_error_reporting = hiera('php_error_reporting'), $php_post_max_size = hiera('php_post_max_size'), ) { package { $packages: ensure => present, }

file {'/etc/php5/apache2/php.ini': ensure => file, content => template("php/php.ini.erb"), notify => Service["apache2"], require => Package["php5"], } }



The template file for a VirtualHost includes variables which will be replaced out by the details for each host.

# ************************************ # Default template for vhosts # Managed by Puppet # ************************************

<VirtualHost *:80> ServerName <%= @vhost_name %> DocumentRoot <%= @vhost_docroot %> <Directory <%= @vhost_docroot %>> Options <%= @vhost_options %> AllowOverride All Order allow,deny allow from all </Directory> ErrorLog /var/log/apache2/<%= @vhost_name %>_error.log LogLevel warn CustomLog /var/log/apache2/<%= @vhost_name %>_access.log combined ServerSignature Off </VirtualHost>



In my YAML file I have configures two sites.

apache_vhosts: site1: vhost_docroot: '/var/www/test_site1' vhost_name: 'site1.dev' vhost_options: 'All' site2: vhost_docroot: '/var/www/test_site2' vhost_name: 'site2.dev' vhost_options: 'All'



Use the hiera_hash function to get my site information and pass it to create_resources.

The create_resources function then calls my host manifest with that hash as the data.

class bootstrap {

# Make sure everything is installed

$sites = hiera_hash('apache_vhosts')

create_resources('apache::vhost',$sites)

$databases = hiera_hash('mysql_db')

create_resources('mysql::db',$databases)

}



This is the manifest that creates the databases I need. define mysql::db(

$db_name = false, $db_user = false, $db_password = false, $root_pw = hiera('mysql_root_password'), ) { exec { "create-${db_name}": unless => "/usr/bin/mysql -u${db_user} -p${db_password} ${db_name}", command => "/usr/bin/mysql -uroot -p${root_pw} -e \"create database ${db_name}; grant all on ${db_name}.* to ${db_user}@localhost identified by '$db_password';\"", require => Exec["set-mysql-password"], } }



To set up a new VM

• git clone

• edit the Vagrantfile for IP address, project name

• edit the config.yaml to create sites and databases

• vagrant up



Your entire development environment can now be described in text files.



Share your environment with your team - they just edit the config.yaml.



Provisioning files and data









The challenge

• Take latest files and database dump from Github

• Deploy the three sites with the current Perch version and add-ons

• Run the upgrade and change any templates as needed

• Produce the db dump with placeholders for Github and a Ruby db template with placeholders for the demo server

• Produce zipped archives for use by the demo server



https://forge.puppetlabs.com/puppetlabs/vcsrepo


https://forge.puppetlabs.com/puppetlabs/vcsrepo


Using the Puppetlabs vcsrepo module to clone a repository.

vcsrepo { '/path/to/repo': ensure => present, provider => git, source => 'git://example.com/repo.git', }



The YAML definition for one of our three demos. It includes the Git repo, local path, database details and Perch license key.

I also detail the Perch Add-ons that need to be installed along with Perch Core.

demo2: repo_uri: 'https://github.com/PerchCMS/perchdemo-swift' vhost_path: '/var/www/perchdemos/demo-swift' db_name: 'db_demo_swift' db_user: 'vagrant' db_password: 'vagrant' key: ‘xxxx-xxxx-xxxx-xxxx‘ sql_path: '/sql/swift_demo.sql' install_addons: demo2_blog: addon_name: 'perch_blog' addon_type: 'apps' demo2_forms: addon_name: 'perch_forms' addon_type: 'apps' demo2_questions: addon_name: 'perch_questions' addon_type: 'apps'



I deploy my databases and sites but also set off a build of the demos.

class bootstrap { $databases = hiera_hash('mysql_db') create_resources(‘mysql::db',$databases)

$demos = hiera_hash('demo_deploy') create_resources(‘perchdemo::deploy',$demos)

$sites = hiera_hash('apache_vhosts') create_resources('apache::vhost',$sites) }



In deploy.pp we get the files from git, each site has a database dump which I do some string replacement on - then import it.

vcsrepo { "${vhost_path}": ensure => present, provider => git, source => $repo_uri, }

exec { "replace-${db_name}": command => "/bin/sed -i 's/{firstname}/REPLACE_firstname/g' ${vhost_path}${sql_path} ; /bin/sed -i 's/{lastname}/REPLACE_lastname/g' ${vhost_path}${sql_path} ; /bin/sed -i 's/{email}/[email protected]/g' ${vhost_path}${sql_path} ; /bin/sed -i 's/{username}/REPLACE_username/g' ${vhost_path}${sql_path} ; /bin/sed -i 's/{password}/5f4dcc3b5aa765d61d8327deb882cf99/g' ${vhost_path}${sql_path}", }

exec { "import-${db_name}": command => "/usr/bin/mysql -uroot -p${root_pw} ${db_name} < ${vhost_path}${sql_path}", }



Up to date Perch Core and add-ons are in a local file store.

I use a Ruby .erb template for the Perch Config so I can add the database details and license key.

I use create_resources to add the add-ons specified in the YAML for this site.

file { "${vhost_path}/public_html/perch/core": ensure => present, source => "${file_store}/core", recurse => true, }

file { "${vhost_path}/public_html/perch/config/config.private.php": ensure => present, content => template('perchdemo/config.private.php.erb'), }

create_resources('perchdemo::copy_addons',$install_addons, {'vhost_path'=>$vhost_path,'file_store'=>$file_store})



At the command line you can run puppet apply to run a manifest.

In this case I am running builder.pp which builds me an archive to upload for each site. > puppet apply --modulepath=/vagrant/

puppet/modules --hiera_config /vagrant/puppet/hiera.yaml -e "include perchdemo::builder"



Taking Puppet to Production



Puppet Masters



Masterless Puppet



You don’t have to provision the entire server using Puppet. Start with small tasks.



Use Puppet to create user accounts with the correct privileges and ssh keys on each server you set up.



Use an inexpensive VPS for client staging sites. Manage the VirtualHosts using Puppet.



https://forge.puppetlabs.com/puppetlabs/apache


https://forge.puppetlabs.com/puppetlabs/apache


Schedule regular Puppet runs to check that services are running, and restart them if not



Puppet can ensure files and directories exist and they have the correct permissions set



Using Puppet can allow people to edit configs without needing privileges on

production servers.



Configuration can be edited, checked into Git and reviewed before being deployed.



Modules you use on the server can often be also used in development. Ensuring the same environment.



Change one small thing.



Improve one thing about your workflow. Build from there.



Thank you FILIVE!

Rachel Andrew


[email protected]

@rachelandrew



mailto:[email protected]

Technology

Configuration Management with Puppet