2014

Presented by

How Puppet Enables the Use of Lightweight Virtualized ContainersJeff McCune

Software Developer | Puppet Labs

@0xEFF

Presented by

What we’ll coverCompare and contrasts a Jenkins+LDAP deployed onto virtual machines and onto lightweight containers.

!

Jenkins and LDAP Puppet Forge Modules Migrating from a shared VM to containers Common Problems with Service Management Summary of Lessons Learned

Presented by

Starting Pointhttps://github.com/jeffmccune/puppetconf2014

Traditional VM

Puppet deploys Jenkins with LDAP security

http://sharedvm:18080

Presented by

Shared VMVagrant VM

OpenLDAPJenkins

Puppet

Puppet configures Jenkins and OpenLDAP

Vagrant Puppet provisioner

Presented by

Migrating to ContainersVagrant VM

OpenLDAPJenkins

Puppet at runtime

Docker Host

!

!

OpenLDAP Container

!

!

Jenkins Container

Puppet at build time

Puppet at build time

Presented by

Puppet Forge module re-useIt is faster to deploy Jenkins with LDAP using forge modules:

rtyler/jenkins module 90% code re-use: 831 lines from the Forge, 88 lines added

camptocamp/openldap module85% code re-use: 584 lines from the Forge, 108 lines added

Presented by

Site Specific CustomizationsInitialize LDAP for dc=jeffmccune,dc=net

Configure LDAP admin account and password

Load initial schema and base OU’s into LDAP

Configure Jenkins to use LDAP security

Presented by

Moving from a VM to a ContainerChallenges in Puppet:

Service resources fail and must be overridden

Downstream resources require the service resource

LDAP initial schema load requires Service[slapd]

Presented by

Why do service resources fail? Two different models of containers:

Those with and without a service management framework.

In either case there is no service management framework running during the image build phase.

Presented by

ContainersTwo Models of Containers:

OS Virtualization Model - Heavy(Solaris Zones, FreeBSD Jails, IBM WPAR’s)

Microservice Model - Light (Docker, CoreOS, Kubernetes)

Presented by

False Dichotomy over “Light”Weight is a range, not a boolean.

Single-process image

Multi-process image

Full-os image

Runtime manageability decreases as processes decrease

Presented by

Single-process MicroservicesIn general:

Executes the application process directly, no init system

startup time comparable to normal process startup

Puppet does not operate at runtime

Difficult to manage with volumes and sidekick processes

Presented by

Multi-process MicroservicesIn general:

Start an init system (SysV init, systemd, upstart)

The service management framework manages services

Runs ~5 or more processes per container

Puppet works at runtime without modification

Presented by

Full OS Virtualization ContainersRun everything a traditional OS would run

Dozens of running processes in the container

Long startup time (> 30 seconds)

Easier to manage at runtime

Better for some situations, e.g. mimicking a full running OS

Presented by

Container ImagesMicroservice containers have a distinct build phase

The result of the build step is an image, ideally immutable

Instances execute from a built base image

The Dockerfile is an example of a build script

Presented by

DockerfileFROM centos:centos7

ADD puppet.tar.gz /

RUN puppet apply -v --modulepath=/puppet/modules \ /puppet/manifests/ldap.pp

EXPOSE 389

CMD ["/usr/sbin/init"]

Presented by

DockerfileFROM centos:centos7

ADD puppet.tar.gz /

RUN puppet apply -v --modulepath=/puppet/modules \ /puppet/manifests/ldap.pp

EXPOSE 389

CMD ["/usr/sbin/init"]

Presented by

DockerfileFROM centos:centos7

ADD puppet.tar.gz /

RUN puppet apply -v --modulepath=/puppet/modules \ /puppet/manifests/ldap.pp

EXPOSE 389

CMD ["/usr/sbin/init"]

Presented by

DockerfileFROM centos:centos7

ADD puppet.tar.gz /

RUN puppet apply -v --modulepath=/puppet/modules \ /puppet/manifests/ldap.pp

EXPOSE 389

CMD ["/usr/sbin/init"]

Presented by

docker build$ docker build --tag base_image . Sending build context to Docker daemon 4.718 MB Step 0 : FROM centos:centos7 ---> 3236320e3562 Step 1 : ADD puppet.tar.gz / ---> bc9aa11de092 Step 2 : RUN puppet apply … ---> Running in efdda7633bfc

Presented by

docker build$ docker build --tag base_image . Sending build context to Docker daemon 4.718 MBStep 0 : FROM centos:centos7 ---> 3236320e3562 Step 1 : ADD puppet.tar.gz / ---> bc9aa11de092 Step 2 : RUN puppet apply … ---> Running in efdda7633bfc

Presented by

docker build$ docker build --tag base_image . Sending build context to Docker daemon 4.718 MB Step 0 : FROM centos:centos7 ---> 3236320e3562 Step 1 : ADD puppet.tar.gz / ---> bc9aa11de092 Step 2 : RUN puppet apply … ---> Running in efdda7633bfc

Presented by

docker build$ docker build --tag base_image . Sending build context to Docker daemon 4.718 MB Step 0 : FROM centos:centos7 ---> 3236320e3562 Step 1 : ADD puppet.tar.gz / ---> bc9aa11de092 Step 2 : RUN puppet apply … ---> Running in efdda7633bfc

Presented by

Docker Build with Puppet Try #1Error: /Service[slapd]: Could not evaluate: Could not find init script for 'slapd'

Warning: Openldap_database[dc=jeffmccune,dc=net]: Skipping because of failed dependencies

Warning: Exec[inetorgperson schema]: Skipping because of failed dependencies

And so on for all dependent resources…

Presented by

systemd is not installedThe error, Could not find init script for ‘slapd’ is caused by a fake systemd in the base centos image

$ docker run -i -t centos rpm -qa | grep systemd fakesystemd-1-15.el7.centos.noarchsystemd-libs-208-11.el7_0.2.x86_64

Presented by

Fix #1 Install the real systemdReplace fakesystemd with the real deal

microservice => virtualized os model

Start a minimum number of services with systemd

New base image: centos7vps with systemd

Presented by

Fix #1 DockerfileFROM centos:centos7

RUN yum -y swap \ -- remove fakesystemd \ -- install systemd systemd-libs

# RUN rm unit files in /{lib,etc}/systemd/system

CMD ["/usr/sbin/init"]

Presented by

Build new base image$ docker build -t centos7vps .

This base image has the real systemd installed.

Presented by

New DockerfileFROM centos7vps # <= was centos:centos7

ADD puppet.tar.gz /

RUN puppet apply -v --modulepath=/puppet/modules \ /puppet/manifests/ldap.pp

EXPOSE 389

CMD ["/usr/sbin/init"]

Presented by

Working for docker run…Puppet works inside a running docker container

systemd is running

Presented by

Working for docker run…Start the systemd container

$ CID=$(docker run --privileged -d centos7vps)

Enter the container

$ sudo /usr/local/bin/docker-enter $CID

Presented by

Working for docker run…Run Puppet:

bash-4.2# puppet apply /puppet/manifests/ldap.pp Compiled catalog for ldap in 1.05 seconds Service[slapd]/ensure: 'stopped' => 'running' Finished catalog run in 30.15 seconds

Presented by

Not working for docker buildsystemd is present, but not running during the build phase

$ docker build -t ldap:vps . …Error: Could not start Service[slapd]: Execution of '/usr/bin/systemctl start slapd'returned 1: Failed to get D-Bus connection: No connection to service manager.

Presented by

Moving aheadWe could stop now and simply run puppet every time the container starts prior to the application starting.

This would increase service start time.

Application images become mutable, managed by Puppet.

Doesn’t fit the microservice model very well.

Presented by

Goal: Puppet configured images Puppet configures immutable pre-configured base application images

Immutable images provide known good state

Closer to the microservice model

Compatible with the os virtualization model

Presented by

Docker build with Puppet Try #2Override the Service[slapd] to avoid systemd at build time

class ldap_override inherits openldap::server::service { Service[slapd] { ensure => running, enable => undef, start => '/usr/sbin/slapd -u ldap -h "ldapi:/// ldap:///" } }

Presented by

Success!We’re able to build an immutable application image fully configured by Puppet.

Presented by

LDAP running in systemd/usr/bin/docker -d --selinux-enabled \_ /usr/sbin/init \_ /usr/lib/systemd/systemd-journald \_ /bin/dbus-daemon --system \_ /usr/sbin/slapd -u ldap -h ldapi:/// ldap:///

Presented by

Moving further aheadWe could stop here, but there are a number of trade-offs

systemd requires the container to run in privileged mode

More processes: systemd, journald, dbus, application

Slower startup / tear-down time (~5 seconds)

Still faster than a VM (~30+ seconds)

Presented by

Final DockerfileFROM centos7vps

ADD puppet.tar.gz /

RUN puppet apply -v /puppet/manifests/ldap.pp

EXPOSE 389

CMD ["/usr/sbin/slapd", "-u", "ldap", \ "-h", "ldapi:/// ldap:///", "-d", "0"]

Presented by

Final DockerfileFROM centos7vps

ADD puppet.tar.gz /

RUN puppet apply -v /puppet/manifests/ldap.pp

EXPOSE 389

CMD ["/usr/sbin/slapd", "-u", "ldap", \ "-h", "ldapi:/// ldap:///", "-d", "0"]

Presented by

docker build ldap:microserviceBuild is required for deployment

$ docker build -t ldap:microservice .

Presented by

Single-process Microservice Only one process running in the container:

/usr/bin/docker -d --selinux-enabled \_ /usr/sbin/slapd -u ldap -h ldapi:/// ldap:/// -d 0

Start / Stop time: ~1 second

Presented by

Puppetized MicroserviceThe application image is immutable

Changes to the application require a new image build

Centralized reporting via Puppet

version control via Puppet code

Re-usable modules from the Puppet forge

Presented by

Putting it togetherLink the jenkins microservice with the ldap microservice

$ docker run -d --name ldap ldap:microservice $ docker run -d --name jenkins \ --link ldap:ldap \ --publish 18080:8080 \ jenkins:microservice

Presented by

ResultsThe application (jenkins) is now isolated

With isolated dependencies included (ldap)

The applications are immutable

Multiple copies can easily be deployed

Difficult to change running instances

Presented by

Making a change to LDAPNeed to modify the LDAP indexes as an optimization

1: Update Puppet Configuration

2: Rebuild LDAP image

3: Re-deploy LDAP

4: Re-deploy Jenkins (The link to LDAP is static)

Presented by

Advantages and DisadvantagesImmutable known good state

Deployment is highly repeatable and consistent

App and Dependency are tightly coupled

Ambassador Pattern decouples services (with tedium)

!

Presented by

SummaryForge modules are re-usable in Docker containers

Service resources pose a challenge

Override the service resource to build immutable images

Multi-process images with systemd are an option

Presented by

Thank you!Code: github.com/jeffmccune/puppetconf2014

David Lutterkort’s Dockercon talk: http://links.puppetlabs.com/lutter-docker

!