Managing Configuration Drift and Auditing with Salt

Duncan Mac-Vicar P.

Director, Data Center Management R&D, SUSE

dmacvicar@suse.com

2

How to manage infrastructure?

#!/bin/bashcat <<EOF |server1server2EOFwhile read line ssh -q user1@${line} zypper updone

Sysadmin Alexis

●Manages his servers with bash scripts.●They reside in`~/bin`●Strict ownership and approval process.

New colleague: Devops Adrian

➔Writes "Configuration Management" recipes and templates➔They reside in `git`.

/etc/motd: file.managed: - source: salt://common/motd

apache: pkg.installed

5

The two brains of IT

Mode 2Mode 1

Agile, DevOpsWaterfall, ITIL

New & Uncertain ProjectsConventional Projects

Short Cycle (days, weeks)Long-cycle Times (months)

AgilityReliability

Devops Adrian explains

“If somebody changes the configuration, I just re-apply it and the tool brings it to the correct state.”

7

Sysadmin Alexis reads:

Configuration management (CM) is a systems engineering process for establishing and maintaining consistency of a product's performance, functional, and physical attributes with its requirements, design, and operational information throughout its life.

Infrastructure as code

➢Has become a pragmatic way to implement configuration management for IT infrastructure.➢We know how to manage change of source code.➢We have the tools and processes for it.

Salt 101

Ports: 4505-4506

Master

Minion

Minion

Minion

ØMQ

Salt 101

configuration commands

results

master

minions

$ salt ‘*’ pkg.install foo

$ salt host1 docker.pause c001

$ salt ‘web*’ cmd.run \

‘cat /etc/fstab’

$ salt ‘*’ state.apply

What is a state?

configuration commands

results

master

minions

/etc/motd:

file.managed:

- source: salt://common/motd

apache:

pkg.installed

“state” is how Salt calls configuration, in its declarative form.

States

Non-Compliant system

$ salt minion1 state.apply test=True

minion1:---------- ID: /etc/motd Function: file.managed Result: None Comment: The file /etc/motd is set to be changed Started: 10:06:05.021643 Duration: 30.339 ms Changes: ---------- diff: --- +++ @@ -1 +1 @@ -Have a lot of fun... +This is my managed motd

Summary for minion1------------Succeeded: 1 (unchanged=1, changed=1)Failed: 0------------Total states run: 1

New trainee

$ useradd -r mudserver

Let’s run it again

$ salt minion1 state.apply test=True

minion1:---------- ID: /etc/motd Function: file.managed Result: None Comment: The file /etc/motd is set to be changed Started: 10:06:05.021643 Duration: 30.339 ms Changes: ---------- diff: --- +++ @@ -1 +1 @@ -Have a lot of fun... +This is my managed motd

Summary for minion1------------Succeeded: 1 (unchanged=1, changed=1)Failed: 0------------Total states run: 1

The change was not detected

It was not part of the configuration.

# We can express...

joe:

user.present

# How to express?

any other:

user.absent

●Devops tools focus in creating new systems.●Not all change accounted. “Is it really Configuration Management”.● What they call "Configuration Management" is really "Automation".●The novelty is more about the declarative approach (vs imperative).

Disappointed Sysadmin

Are “Classic” IT priorities the same?

●Detecting Configuration Drift.●Auditing Compliance.●Documenting infrastructure. Drift

Audit

Document

“Incomplete” Configuration

Baseline

States Templates

Where is the baseline?

In configuration management, a "baseline" is an agreed description of the attributes of a

product, at a point in time, which serves as a basis for defining change. ~~MIL-HDBK-61

How to define a baseline?How to integrate it with the rest of the

configuration?

Snapper (http://snapper.io)

➔snapper is to snapshots what zypper/apt-get/dnf are to packages.➔First demoed in SUSECon 2011!➔Main feature of SUSE Linux Enterprise 12

Created by SUSE, available everywhere

●https://wiki.archlinux.org/index.php/Snapper●https://apps.fedoraproject.org/packages/snapper●https://packages.debian.org/search?keywords=snapper

(don't forget to mention btrfs)

Snapper 101

snapper list-configs

snapper list

snapper create

snapper mount <number>

snapper status <number1>..<number2>

snapper diff <number1>..<number2> [files]

snapper undochange <number1>..<number2> [files]

YaST and zypper take snapshots automaticallyIn grub menu you can boot old snapshots

YaST2 snapper

Salt and Snapper integration I

configuration commands

results

master

minions

salt '*' snapper.list_snapshots

salt '*' snapper.get_snapshot

salt '*' snapper.create_snapshot

salt '*' snapper.undo

salt '*' snapper.diff

Salt andSnapper Integration

$ salt minion2 snapper.run function=file.append args='["/etc/motd", "some text"]'

minion2:

Wrote 1 lines to "/etc/motd"

...

pre | 21 | | Thu.. | root | number | salt job 6668 | salt_jid=6668

post | 22 | 21 | Thu... | root | number | salt job 6668 | salt_jid=6668

Salt and Snapper integration

$ salt minion2 snapper.diff_jid 6668

minion2:

----------

/etc/motd:

--- /.snapshots/21/snapshot/etc/motd

+++ /.snapshots/22/snapshot/etc/motd

@@ -1 +1,2 @@

Have a lot of fun...

+some text

snapper.undo_jid also works

State module

Back to the baseline problem, imagine you could say:

# Starting from snapshot #3baseline: - ????# then ...

/etc/motd: file.managed: - source: salt://common/motd

apache: pkg.installed

Baseline

States Templates

State module

You can!

my_baseline:

snapper.baseline_snapshot:

- number: 20

- ignore:

- /var/log

- /var/cache

/etc/motd:

file.managed:

- source: salt://common/motd

Baseline

States Templates

If the somebody adds a new user, a drift against the baseline rule will happen:

$ salt minion1 state.apply test=Trueminion1:---------- ID: my_baseline Function: snapper.baseline_snapshot Result: None Comment: 1 files changes are set to be undone ... Changes: ... /etc/passwd: ... diff: --- /etc/passwd +++ /.snapshots/21/snapshot/etc/passwd @@ -22,5 +22,3 @@ duncan:x:1000:100:Duncan Mac-Vicar P.:/home/duncan:/bin/zsh -mudserver:x:167:100::/home/mudserver:/bin/bash---------- ID: /etc/motd...Succeeded: 2 (unchanged=2, changed=2)

If you apply the state (eg. no `test=True`), the system will be set to the state of the baseline snapshot before applying the rest of the states.

Applying states

$ salt minion1 state.apply

Current

Baseline state

... states

Managing snapshots by number?

$ salt '*' snapper.create_baseline Creates a snapshot and adds a “baseline” tag to the “userdata” property of each snapshot.

Type # Pre Date Desc userdata

single 0 current

...

post 20 19 Sept 26... important=no

...

single 22 Sept 26... baseline_tag=baseline

Baseline tags

●You can move the baseline, without affecting your state.●The last tagged snapshot will be used.

last_production: snapper.baseline_snapshot: - tag: baseline

ConfigureD

riftDocum

ent

Bas

eline

Audit

Salt Snapper module

●Already submitted upstream. Will be part of Carbon release.●Also available in SUSE Linux Enterprise/SUSE Manager 2015.8.x Salt package●Carbon also supports automatic snapshots when applying states

https://docs.saltstack.com/en/develop/topics/releases/carbon.html

(about state snapshots)

$ salt minion2 snapper.run function=state.apply

Other Resources to Manage Drift

Salt Survey Runner Module

Survey groups the returned values in pools of unique results.

salt-run survey.diff survey_sort=up "*" cmd.run 'cat /etc/hosts'

This tells you which server differs from the others.

v1/etc/hosts

v2/etc/hosts

Salt Package Module

salt 'web*' pkg.diff /etc/sudoers

Tells the difference between the `/etc/sudoers` of the original package vs the installed one.

Hubble (http://hubblestack.io)

Tool Purpose

Nova Auditing Framework

Pulsar File integrity monitor, security events

Nebula Query infrastructure security snapshots

Quasar Reporting

Available Nova modules

● grep (configuration values)● iptables (firewall rules)● netstat (listening ports)● openscap (CVE scan)● openssl (cert validation & expiration)● pkg (installed packages)● service (running services)● stat (ownerships & permissions)● sysctl (kernel parameters)● vulners.com (CVE scan)

Future work

44

The two brains of IT

Mode 2Mode 1

Agile, DevOpsWaterfall, ITIL

New & Uncertain ProjectsConventional Projects

Short Cycle (days, weeks)Long-cycle Times (months)

AgilityReliability

45

“Bimodal Datacenter”

DeploymentHigh

AvailabilityScaling MonitoringNetworking

Software Defined *:ComputeStorageNetwork

Magnum

Containers

Mode 1 Mode 2

46

Docker images

47

●You bought into the hype and automated everything with Salt.

● You have Salt states and templates for all your infrastructure. salt://happy

● Now you wand to deploy an app into a container and you need a container image...

Motivation

48

Dockerfiles

FROM ubuntu:12.04

RUN zypper -n in foo

RUN echo "key=val" > /etc/foo/config

49

Building images

foo:

pkg.installed

/etc/foo/config:

file.managed:

- source: salt://myapp/foo.config

salt mybuilder0 dockerng.state myapp mods=myapp

50

Benefits

Reuse states and templates across workloads. Access templates, pillar data (eg. secrets) directly from Salt. Access to Salt modules for configuration.

Implemented as ability to run arbitrary Salt modules inside running containers.

Auditing

51

Thanks for listening

53