chefconf 2015 intro to analytics - Viewdeck - Home · 2016-03-22 · Web Servers Application Servers Database. Exponential Increase in Size Leads to ... • Success/Failure status

v 3.0.0

ChefConf 2015Introduction to Chef Analytics Platform

v 3.0.0

PrerequisitesHave an ssh client Have a good text editor (Atom, Sublime, vim, emacs) Have ChefDK (latest) installed Git & GitHub Account (Optional) Chef Fundamentals Training (or equivalent experience)

v 3.0.0

Chef Development Kit Version: 0.4.0

Confirm your setup$ chef -v

v 3.0.0

S C O T T F O R D

[email protected] smford22smford22

@sford422

[email protected]

S O L U T I O N S E N G I N E E R

v 3.0.0

L A R R Y

E I C H E N B A U M

[email protected] larryebaumlarryebaum

@larryebaum

[email protected]

v 3.0.0

DISCUSSIONIntroduce Yourselves

Name

Current job role

Previous job roles/background

Experience with Chef and/or config management

Favorite Text Editor

6

v 3.0.0

Agenda

Introduction to the Chef Analytics Platform

Workshop Environment Setup

Installing the Analytics Platform

Using the Analytics Platform

v 3.0.0

Course Objectives & Style

v 3.0.0

Course ObjectivesAfter completing this course you will be able to:

Install Chef Server and Chef Analytics using Chef Provisioning Understand the Chef Analytics platform including: • chef-client --audit-mode • Actions • Controls • Rules • Notifications

Write compliance controls recipes for your infrastructure, and use rules to and notifications to be alerted when tests succeed or fail

v 3.0.0

Training is a discussionLots of hands on labs Lots of typing Ask questions when they come to you Ask for help when you need it Help each other We will troubleshoot and fix bugs on the spot

v 3.0.0

Just an IntroductionToday is just an Introduction to the Chef Analytics Platform We’ll cover lots of topics but won’t go too deep on any of them We will have a Q&A with some of the Engineers from the Analytics Team at the end of the workshop

v 3.0.0

Login to Workshop HipChat Channel

v 3.0.0

Login to HipChatFor the purpose of this class, and in an effort to work together as a team, we have setup a public HipChat channel for all of us to use. You will need to create an account if you do not have one, but you can find that link here…

http://bit.ly/1D3bs4i

http://bit.ly/1D3bs4i

v 3.0.0

HipChat Quick Tips• @all notifies everyone regardless if they are present or not

• @here notifies everyone currently in the room

• @username to address a specific person in the room

• /code <paste code> allows you to paste code snippets in the room

v 3.0.0

Get logged in and say hello!

v 3.0.0

CheckpointAt this point you should have: • ChefDK • A programmers text editor • SSH client • Logged into workshop HipChat channel

v 3.0.0

Introduction to the Analytics Platform

The promise of the coded business

Transformation to high-velocity

Regulatory compliance frameworks

OFAC USA PATRIOT Act Gramm-Leach-Bliley Act Red Flags Rule

Bank Secrecy Act Sarbanes-Oxley Regulation E Dodd-Frank

False Claims Act HIPAA European Central Bank regulations Prudential Regulation Authority

Financial Conduct Authority HITECH PCI DSS

http://www.treasury.gov/about/organizational-structure/offices/Pages/Office-of-Foreign-Assets-Control.aspx

http://www.fincen.gov/statutes_regs/patriot/

http://www.business.ftc.gov/privacy-and-security/gramm-leach-bliley-act

http://www.business.ftc.gov/privacy-and-security/gramm-leach-bliley-act

http://www.business.ftc.gov/privacy-and-security/red-flags-rule

http://www.fincen.gov/statutes_regs/bsa/

http://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act

http://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act

http://www.federalreserve.gov/bankinforeg/regecg.htm

http://www.sec.gov/spotlight/dodd-frank.shtml

http://www.justice.gov/sites/default/files/civil/legacy/2011/04/22/C-FRAUDS_FCA_Primer.pdf

http://www.hhs.gov/ocr/privacy/

https://www.ecb.europa.eu/ecb/legal/ssm/framework/html/index.en.html

http://www.bankofengland.co.uk/pra/Pages/default.aspx

http://www.bankofengland.co.uk/pra/Pages/default.aspx

http://www.fca.org.uk/firms/being-regulated/meeting-your-obligations

http://www.hhs.gov/ocr/privacy/hipaa/administrative/enforcementrule/hitechenforcementifr.html

https://www.pcisecuritystandards.org/security_standards/index.php

The conflict between compliance and velocity

The compliance challenge

The velocity challenge

Reconciling compliance and velocity

Analyze

• Be clear about what the desired system outcome actually is

• Take regulatory requirements and enterprise policies into account

• Choosing the desired state and expressing it at an appropriate level of detail can be more challenging problems than writing the automation code itself!

Specify

• Closing the gap between specifying and implementing regulations requires an unambiguous expression of the requirement in human- and machine-readable form.

• A domain-specific formal language (DSL) can achieve this level of clarity and precision.

• Chef recipes, tests and compliance rules are ideal for the task.

Examplepackage 'apache2'

service 'apache2' do action [:start, :enable] end

Test

• Automated tests give confidence that the requirement has actually been met

• Writing the tests first give developers and system administrators a clear set of standards that must be met for compliant systems.

• Automated tests scale better than manual tests.

Example

Certify

• A separate certification step is not always required • In some cases, regulatory requirements or

organizational processes do require a final human sign off

• The better your tests, the shorter the certification step can be

• Be sure not to confuse certification and testing

The changing role of the compliance officer

A single accelerated cycle

1980

1981

1982

1983

1984

1985

1986

1987

1988

1989

1994 1999 2004 2009 2012 2013 2014 2015 2016 2017 2018 2019

Enterprises Have Nearly Unlimited Computing Resources

Virtual NodesPhysical Hardware

1980Mainframe

1990Client/Server

2000Datacenter

2010+Web-Scale

20

40

60

80

100

120Mill

ions

Mill

ions

of S

erve

rs

Exponential Increase in Size Leads toOperational Complexity

Web Servers

Application Servers

Database

Exponential Increase in Size Leads toOperational Complexity

Web Servers

Application Servers

Database

Add 1 server20+ Changes

12+ New Dependences

Speed of Execution Requires Visibility

• Change tracking • Security logs • Auditing • Performance monitoring

Chef Analytics Keeps Your Finger on the Pulse of Your Infrastructure

Chef Analytics Provides Three Core Components

• Actions and Run History • Record any policy or administrative changes to any object managed by Chef Server

• Track changes through all sources including management console, knife command or direct application of chef-client

• Real-time Reporting • Browse events in a friendly web UI with search, filters and sorting options

• Integrate with existing tools via API • Notifications

• Alert teams of every change through built-in messaging and email integration

• Extend notifications to existing systems with simple webhook architecture

Chef Analytics History

• Launched in May 2014 • Builds on Reporting (Run history) feature shipped in 2013

• Adds new fundamental data collection components • Actions – track policy modifications on the Chef Server

• Compliance – assert controls on changes to infrastructure and policy

• Pluggable analytics pipeline

• Chef Actions component available now!

Chef Analytics Architecture

Chef Analytics Data Flow

Reporting - Who did what on your Chef Server?

• Single view of what is changing in your infrastructure • Success/Failure status of individual Chef Client runs • Rollups of success/failure counts • Rollups of run durations

• Drill-down detail to individual resource convergence • State before/after

• Diffs (e.g. for templates, files)

• Errors

Reporting – what’s happening on chef-client runs ?

Actions – The Real-time Event Stream

• Provide a read-only view of what happened

• Can be customized to meet audit and compliance reporting requirements

• Allow administrators to react to events as they happen or after the fact investigation

• “What happened just before nodes started failing runs?”

• “When did our systems gets patched for Heartbleed?”

Controls

• A control is an automated test that is built into a cookbook

• Can be used to test the state of the system for compliance

• Wrapper around ServerSpec testing framework • Can be used to create audit tests around compliance frameworks such as PCI, HIPAA, and Sarbanes-Oxley

Rules and Notifications

• Create specialized rule sets to match the output of audits in your infrastructure

• Send notifications • Integrate with HipChat • Send email alerts • Web hooks to api endpoints • Integrations with third party solutions like Splunk

v 3.0.0

Workshop Environment Setup

v 3.0.0

Objectives

• Understand the different configuration options when installing Chef Analytics

• Use Chef Provisioning to deploy a Chef Server & Analytics Server in AWS

• Login to Chef Server and Analytics • Setup your workstation to manage infrastructure with Chef • Bootstrap a node to your Chef server and configure it with Chef

v 3.0.0

Installing Chef Analytics

Prerequisites• An x86_64 compatible system architecture:

• RHEL/CentOS (5.x, 6.x) • Ubuntu (10.04, 12.04, 13.04)

• Chef server version 12.0.3 or Enterprise Chef version 11.3 • chef-client version 12.1 is required for audit-mode • The Chef management console must be installed on the Chef server prior to

installing Chef analytics • Chef reporting is installed on the Chef server • A resolvable hostname that is specified using a FQDN or an IP address • A local mail transfer agent that allows the Chef server to send email notifications • A connection to NTP to prevent clock drift

Chef Analytics Data FlowAnalytics maps 1:1 with a Chef Server

Supported Configurations

• Standalone • Everything installed on one server • Configured to a Chef Server

• Tiered • 1 Backend server • Multiple front end servers behind a load balancer • Configured to a Chef Server

v 3.0.0

LOCAL

# Install Chef Server

- Stand up a instance to install on

- SSH to the instance

- Download the chef-server-12 package

- Install the package

- Create a /etc/opscode/chef-server.rb config file with <FQDN>

- Configure the server with ‘chef-server-ctl reconfigure’

- Install opscode-manage package

- Run opscode-manage-ctl reconfigure

- Install opscode-reporting

- Run opscode-reporting-ctl reconfigure

- Install opscode-analytics package

- Run opscode-analytics-ctl reconfigure

- Run chef-server-tcl reconfigure

Installation Steps - Chef Server

http://docs.chef.io/server/install_server.html


v 3.0.0

LOCAL

# Install Chef Analytics

- Stand up a instance to install on

- SSH to the instance

- Download the chef-analytics package

- Install the package

# Configure Chef Server with Analytics server

- Configure oc_id and RabbitMQ remote access /etc/opscode/chef-server.rb

- run chef-server-ctl reconfigure

- scp /etc/opscode-analytics to the Analytics host

# Configure Analytics

- Add analytics FQDN to /etc/opscode-analytics/opscode-analytics.rb

- run opscode-analytics-ctl reconfigure

Installation Steps - Chef Analytics



OR…( maybe there’s a better way )

v 3.0.0

Chef Provisioning

Chef Provisioning

• Originally released as Chef Metal at ChefConf 2014 • Allows creation of instances in Chef Recipes • Orchestrate complex deployment of applications • Moves more towards “Infrastructure as Code”

https://github.com/chef/chef-provisioning


v 3.0.0

LOCAL

require 'chef/provisioning/aws_driver' with_driver 'aws'

machine "scott-test" do machine_options( :ssh_username => 'root', :image_id => 'ami-b6bdde86', :bootstrap_options => { :instance_type => 't1.micro', :key_name => 'chefconf2015' } ) end

machine resource



v 3.0.0

LOCAL

require 'chef/provisioning/aws_driver'

with_driver 'aws'

with_machine_options( bootstrap_options: { instance_type: 't1.micro', key_name: 'chefconf', security_group_ids: 'default' }, ssh_username: 'root', image_id: 'ami-a9de9c99' )

chef provisioning drivers



v 3.0.0

LOCAL

•machine•machine_batch•machine_execute•machine_file•machine_image

Chef Provisioning • AWS• FOG• Azure• Vagrant• vsphere• Docker, LXC• Hanlon, OpenCrowBar• More coming….



v 3.0.0

LOCAL

require 'chef/provisioning'

machine_batch do

machines %w(primary secondary web1 web2)

end

machine 'primary' do

recipe 'initial_ha_setup'

end

machine 'secondary' do

recipe 'initial_ha_setup'

end

machine_batch do

%w(primary secondary).each do |name|

machine name do

recipe 'rest_of_my_configuration'

end

end

Chef Provisioning Recipe



v 3.0.0

Legend

Workshop Virtual Machines

• Each student will be provided: • 1 Node to use for Chef Provisioning • 1 Node to manage with Chef

http://bit.ly/1DlXOYD

http://bit.ly/1DlXOYD

v 3.0.0

REMOTE

Three different places to run commands

$ ssh chef@<EXTERNAL-IP-ADDRESS>This is an example of a command you run on your workstation

[chef@provisioning-node ~]$ whoami

[chef@hostname ~]$ whoami

This is an example of a command you run on your provisioning node

This is an example of a command you run on your target node

v 3.0.0

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384

options=3<RXCSUM,TXCSUM>

inet6 ::1 prefixlen 128

inet 127.0.0.1 netmask 0xff000000

inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1

nd6 options=1<PERFORMNUD>

gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280

stf0: flags=0<> mtu 1280

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

ether 78:31:c1:c8:de:92

inet6 fe80::7a31:c1ff:fec8:de92%en0 prefixlen 64 scopeid 0x4

inet 192.168.1.71 netmask 0xffffff00 broadcast 192.168.1.255

nd6 options=1<PERFORMNUD>

Example of a Terminal command$ ifconfig

v 3.0.0

Example of editing a file on your workstation

file "hello.txt" do content "Hello, world!"end

Open in editor: ~/hello.rb

The file named "hello.txt" is created with the content "Hello, world!".

v 3.0.0

Login to Provisioning Node

v 3.0.0

[email protected]'s password:Last login: Mon Mar 16 22:43:58 2015 from 99.8.186.151[chef@provisioning-node ~]$

SSH to your Provisioning Node$ ssh chef@<EXTERNAL-IP-PROVISIONING-NODE>

v 3.0.0

[default]region = us-west-2aws_access_key_id = <ACCESS KEY PROVIDED>aws_secret_access_key = <SECRET KEY PROVIDED>

Setup AWS Credentials[chef@provisioning-node ~]$ cat ~/.aws/config

v 3.0.0

total 8-r--------. 1 chef chef 1675 Mar 20 18:00 chefconf2015-r--------. 1 chef chef 394 Mar 20 18:00 chefconf2015.pub-rw-r--r--. 1 chef chef 0 Mar 20 20:29 known_hosts

SSH Keys Configured[chef@provisioning-node ~]$ ls -l .ssh

v 3.0.0

Initialized empty Git repository in /home/chef/analytics-cluster/.git/

remote: Counting objects: 29, done.

remote: Compressing objects: 100% (23/23), done.

remote: Total 29 (delta 2), reused 29 (delta 2), pack-reused 0

Unpacking objects: 100% (29/29), done.

Git clone the analytics-cluster cookbook[chef@provisioning-node ~]$ git clone https://github.com/opscode-cookbooks/analytics-cluster.git

https://github.com/opscode-cookbooks/analytics-cluster.git

https://github.com/opscode-cookbooks/analytics-cluster.git

v 3.0.0

{ "name": "<firstname-lastname>", "description": "An example environment for a Chef Server and Chef Analytics", "json_class": "Chef::Environment", "chef_type": "environment", "override_attributes": { "analytics-cluster": { "id": "<firstname-lastname>", "aws": { "key_name": "chefconf2015", "ssh_username": "root", "image_id": "ami-cda985fd", "subnet_id": "subnet-8f847cd6", "security_group_ids": "sg-63291806", "use_private_ip_for_ssh": false }, "chef-server": { "flavor": "m3.medium", "organization": "chefadmin" }, "analytics": { "flavor": "m3.medium" } } }

Create an Environment file [chef@provisioning-node ~]$ cd analytics-cluster

[chef@provisioning-node analytics-cluster ]$ cat environments/<firstname-lastname>.json

v 3.0.0

LOCAL

environments/<firstname-lastname>.json{ "name": "<firstname-lastname>", "description": "An example environment for a Chef Server and Chef Analytics", "json_class": "Chef::Environment", "chef_type": "environment", "override_attributes": { "analytics-cluster": { "id": "<firstname-lastname>", "aws": { "key_name": "chefconf2015", "ssh_username": "root", "image_id": "ami-cda985fd", "subnet_id": "subnet-8f847cd6", "security_group_ids": "sg-63291806", "use_private_ip_for_ssh": false }, "chef-server": { "flavor": "m3.medium", "organization": "chefadmin" }, "analytics": { "flavor": "m3.medium" } } }}

http://bit.ly/1Gub7Ha

http://bit.ly/1Gub7Ha

v 3.0.0

source 'https://rubygems.org'

gem 'berkshelf', '~> 3.2'

gem 'chef', '~> 12.0'

gem 'chef-provisioning', git: 'https://github.com/chef/chef-provisioning.git',

ref: 'master'

gem 'chef-provisioning-aws', git: 'https://github.com/chef/chef-provisioning-aws.git',

ref: 'master'

Run bundle install[chef@provisioning-node ~]$ bundle install

v 3.0.0

Updating https://github.com/chef/chef-provisioning.git

Updating https://github.com/chef/chef-provisioning-aws.git

Fetching gem metadata from https://rubygems.org/........

Fetching additional metadata from https://rubygems.org/..

Resolving dependencies...

Using rake 10.4.2

Using addressable 2.3.7

Using builder 3.2.2

Using gyoku 1.2.3

Using mini_portile 0.6.2

Using nokogiri 1.6.6.2

Using akami 1.2.2

Using json 1.8.2

Using aws-sdk-v1 1.63.0

Using multipart-post 2.0.0

Using faraday 0.9.1

…

Run bundle install[chef@provisioning-node ~]$ bundle install

v 3.0.0

source 'https://supermarket.chef.io'

metadata

cookbook 'chef-server-12',

path: 'vendor/chef-server-12'

cookbook 'chef-server-ingredient',

git: 'https://github.com/opscode-cookbooks/chef-server-ingredient.git',

branch: ‘master’

Berksfile[chef@provisioning-node]$ cat Berksfile

v 3.0.0

Resolving cookbook dependencies...Fetching 'analytics-cluster' from source at .Fetching 'chef-server-12' from source at vendor/chef-server-12Using chef-server-ingredient (0.3.0) from [email protected]:opscode-cookbooks/chef-server-ingredient.git (at master)Using chef-server-12 (0.1.3) from source at vendor/chef-server-12Using analytics-cluster (0.1.0) from source at .Using packagecloud (0.0.17)Vendoring analytics-cluster (0.1.0) to cookbooks/analytics-clusterVendoring chef-server-12 (0.1.3) to cookbooks/chef-server-12Vendoring chef-server-ingredient (0.3.0) to cookbooks/chef-server-ingredientVendoring packagecloud (0.0.17) to cookbooks/packagecloud…

Run bundle exec berks vendor cookbooks[chef@provisioning-node ~]$ bundle exec berks vendor cookbooks

v 3.0.0

Starting Chef Client, version 12.0.3[2015-03-18T08:15:38-07:00] WARN: Run List override has been provided.[2015-03-18T08:15:38-07:00] WARN: Original Run List: [][2015-03-18T08:15:38-07:00] WARN: Overridden Run List: [recipe[analytics-cluster::setup_chef_server]]resolving cookbooks for run list: ["analytics-cluster::setup_chef_server"]Synchronizing Cookbooks: - analytics-cluster - chef-server-12 - chef-server-ingredient - packagecloudCompiling Cookbooks...Converging 10 resourcesRecipe: analytics-cluster::setup_chef_server * machine[chef-server-<environment>] action converge ... ...

+node_name 'chefadmin' +chef_server_url 'https://54.148.24.47/organizations/chefadmin' +client_key '/Users/scottford/chef-repo/cookbooks/analytics-cluster/.chef/../.chef/analytics-cluster-data/chefadmin.pem' +cookbook_path '/Users/scottford/chef-repo/cookbooks/analytics-cluster/.chef/local-mode-cache/cache/cookbooks' +trusted_certs_dir '/Users/scottford/chef-repo/cookbooks/analytics-cluster/.chef/trusted_certs'* execute[upload all cookbooks] action run - execute knife cookbook upload --all --cookbook-path /Users/scottford/chef-repo/cookbooks/analytics-cluster/.chef/local-mode-cache/cache/cookbooks[2015-03-18T08:35:47-07:00] WARN: Skipping final node save because override_runlist was given

Running handlers:Running handlers completeChef Client finished, 9/10 resources updated in 730.648963 seconds

Run analytics-cluster::setup_chef_server[chef@provisioning-node ~]$ bundle exec chef-client –z -o analytics-cluster::setup_chef_server –E <firstname-lastname>

v 3.0.0

Starting Chef Client, version 12.0.3[2015-03-18T08:15:38-07:00] WARN: Run List override has been provided.[2015-03-18T08:15:38-07:00] WARN: Original Run List: [][2015-03-18T08:15:38-07:00] WARN: Overridden Run List: [recipe[analytics-cluster::setup_analytics]]resolving cookbooks for run list: ["analytics-cluster::setup_analytics"]Synchronizing Cookbooks: - analytics-cluster - chef-server-12 - chef-server-ingredient - packagecloudCompiling Cookbooks...Converging 10 resourcesRecipe: analytics-cluster::setup_chef_server * machine[analytics-<environment>] action converge ... ... * chef_server_ingredient[opscode-analytics] action reconfigure * execute[opscode-analytics-reconfigure] action run - execute opscode-analytics-ctl reconfigure

Running handlers:Running handlers completeChef Client finished, 11/13 resources updated in 241.197843043 seconds- run 'chef-client -l auto' on analytics-server-scott-ford[2015-03-18T08:43:46-07:00] WARN: Skipping final node save because override_runlist was given

Running handlers:Running handlers completeChef Client finished, 5/5 resources updated in 458.067242 seconds

Run analytics-cluster::setup_analytics[chef@provisioning-node ~]$ bundle exec chef-client –z -o analytics-cluster::setup_analytics –E <firstname-lastname>

v 3.0.0

Show Analytics Public Hostname[chef@provisioning-node ~]$ knife node show analytics-server-<firstname>-<lastname> -a ec2.public_hostname

v 3.0.0

Lab Gist1. Logon to provisioning node 2. clone analytics-cluster cookbook 3. bundle install 4. bundle exec berks vendor cookbooks 5. create environment file 6. bundle exec chef-client -z -o analytics-cluster::setup_chef_server -E <firstname-

lastname> 7. bundle exec chef-client -z -o analytics-cluster::setup_analytics -E <firstname-

lastname>

http://bit.ly/1HZu9Hi

http://bit.ly/1HZu9Hi

v 3.0.0

CheckpointAt this point you should have: • ChefDK • A programmers text editor • SSH client • Logged into workshop HipChat channel • Chef Server & Chef Analytics

v 3.0.0

Setup your Chef Repo

v 3.0.0

Setup your Org on the Chef Server•Create an account •Setup an organization

•Download your Starter Kit

•Setup chef-repo on your workstation

v 3.0.0

Exercise: Set up a working directoryMake a working directory on your laptop under your home directory called ‘~/analytics-workshop', i.e.

Windows:- C:\Users\you\analytics-workshop

Mac/*nix:- /Users/you/analytics-workshop

Navigate to this working directory

v 3.0.0

WARNING: Certificates from 52.123.22.235 will be fetched and placed in your trusted_cert directory (/Users/scottford/analytics-workshop/chef-repo/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should verify the authenticity of these certificates after downloading.

Adding certificate for 52.123.22.235 in /Users/scottford/analytics-workshop/chef-repo/.chef/trusted_certs/52.123.22.235.crt

Exercise: use knife ssl fetch$ knife ssl fetch

v 3.0.0

<your-org>-validator.pem

Exercise: Test your workstation$ knife client list

v 3.0.0

Lab Gist1. Logon to your Chef Server 2. Download Starter Kit 3. Create a working directory ~/chef-repo 4. Use knife ssl fetch to pull in self-signed certs 5. Validate config

http://bit.ly/1CrAczG

http://bit.ly/1CrAczG

v 3.0.0

Bootstrap A Node

v 3.0.0

uvo164727i3mvh1jup2.vm.cld.sr --2014-05-13 04:31:10-- https://www.opscode.com/chef/install.sh

uvo164727i3mvh1jup2.vm.cld.sr Resolving www.opscode.com... 184.106.28.90

uvo164727i3mvh1jup2.vm.cld.sr Connecting to www.opscode.com|184.106.28.90|:443... connected.

uvo164727i3mvh1jup2.vm.cld.sr HTTP request sent, awaiting response... 200 OK

uvo164727i3mvh1jup2.vm.cld.sr Length: 15934 (16K) [application/x-sh]

uvo164727i3mvh1jup2.vm.cld.sr Saving to: `STDOUT'

uvo164727i3mvh1jup2.vm.cld.sr

100%[======================================>] 15,934 --.-K/s in 0s


uvo164727i3mvh1jup2.vm.cld.sr 2014-05-13 04:31:10 (538 MB/s) - written to stdout [15934/15934]


uvo164727i3mvh1jup2.vm.cld.sr Downloading Chef 11.8.2 for el...

uvo164727i3mvh1jup2.vm.cld.sr downloading https://www.opscode.com/chef/metadata?v=11.8.2&prerelease=false&nightlies=false&p=el&pv=6&m=x86_64

uvo164727i3mvh1jup2.vm.cld.sr to file /tmp/install.sh.41533/metadata.txt

uvo164727i3mvh1jup2.vm.cld.sr trying wget...

uvo164727i3mvh1jup2.vm.cld.sr url https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-11.8.2-1.el6.x86_64.rpm

...

Exercise: Bootstrap the target node$ knife bootstrap <external address> --sudo –x chef –P chef –N "node1"

v 3.0.0

$ git clone https://github.com/opscode-cookbooks/chef-client.git

Exercise: Use git to clone the chef-client cookbook

$ cd chef-repo/cookbooks

Cloning into 'chef-client'... remote: Counting objects: 2876, done. remote: Total 2876 (delta 0), reused 0 (delta 0), pack-reused 2876 Receiving objects: 100% (2876/2876), 601.53 KiB | 0 bytes/s, done. Resolving deltas: 100% (1405/1405), done. Checking connectivity... done.

https://github.com/opscode-cookbooks/chef-client.git

v 3.0.0

Download chef-client cookbook from github.com

https://github.com/opscode-cookbooks/chef-client

1. click Download ZIP

2. Unzip into chef-repo/cookbooks/

3. cd into that directory

http://github.com

https://github.com/opscode-cookbooks/chef-client

v 3.0.0

$ berks install

Exercise: Use Berkshelf to install dependencies

$ cd chef-repo/cookbooks/chef-client

Resolving cookbook dependencies... Fetching 'chef-client' from source at . Fetching cookbook index from https://supermarket.chef.io... Using apt (2.7.0) Using chef-client (4.2.4) from source at . Using build-essential (2.2.1) Using chef_handler (1.1.6) Using cron (1.6.1) Using logrotate (1.9.1) Using runit (1.5.18) Using windows (1.36.6) Using yum (3.5.3) Using yum-epel (0.6.0)

v 3.0.0

Uploaded apt (2.7.0) to: 'https://52.123.22.235:443/organizations/analytics-workshop' Uploaded build-essential (2.2.1) to: 'https://52.123.22.235:443/organizations/analytics-workshop' Uploaded chef-client (4.2.4) to: 'https://52.123.22.235:443/organizations/analytics-workshop' Uploaded chef_handler (1.1.6) to: 'https://52.123.22.235:443/organizations/analytics-workshop' Uploaded cron (1.6.1) to: 'https://52.123.22.235:443/organizations/analytics-workshop' Uploaded logrotate (1.9.1) to: 'https://52.123.22.235:443/organizations/analytics-workshop' Uploaded runit (1.5.18) to: 'https://52.123.22.235:443/organizations/analytics-workshop' Uploaded windows (1.36.6) to: 'https://52.123.22.235:443/organizations/analytics-workshop' Uploaded yum (3.5.3) to: 'https://52.123.22.235:443/organizations/analytics-workshop' Uploaded yum-epel (0.6.0) to: 'https://52.123.22.235:443/organizations/analytics-workshop'

Exercise: Use Berkshelf to upload chef-client and dependencies

$ berks upload --ssl-verify=false

https://52.10.184.183:443/organizations/analytics-workshop'

v 3.0.0

Create a base role

name "base"description "A base role"run_list "recipe[chef-client]", "recipe[chef-client::config]"default_attributes( "chef_client" => { "config" => { "ssl_verify_mode" => ":verify_peer", } })

Open in editor: chef-repo/roles/base.rb

http://bit.ly/1GHODmd

http://bit.ly/1GHODmd

v 3.0.0

Updated Role base!

Exercise: Upload base role$ knife role from file base.rb

v 3.0.0

node1:

run_list:

role[base]

Exercise: Set run list for managed node$ knife node run_list set node1 'role[base]'

v 3.0.0

Starting Chef Client, version 11.12.8

resolving cookbooks for run list: ["chef-client::delete_validation", "chef-client"]

Synchronizing Cookbooks:

- chef-client

Compiling Cookbooks...

...

Exercise: Run chef-clientchef@node1$ sudo chef-client

v 3.0.0

root 8933 0.3 2.2 130400 37816 ? Sl 03:19 0:01 /opt/chef/embedded/bin/ruby /usr/bin/chef-client -d -c /etc/chef/client.rb -L /var/log/chef/client.log -P /var/run/chef/client.pid -i 1800 -s 300

Exercise: Verify chef-client is runningchef@node1$ ps awux | grep chef-client

v 3.0.0

Lab Gist1. Bootstrap your node 2. Pull down chef-client cookbook 3. Use Berkshelf to pull down cookbook dependencies 4. Use Berkshelf to upload chef-client cookbook 5. Create a base role 6. Update your node’s run_list to use the base role 7. Re-run chef-client

http://bit.ly/1CrABSA

http://bit.ly/1CrABSA

v 3.0.0

CheckpointAt this point you should have: • ChefDK • A programmers text editor • SSH client • Logged into workshop HipChat channel • Chef Server & Chef Analytics • A node bootstrapped with a base role

v 3.0.0

Using the Analytics Platform

v 3.0.0

Objectives

• Understand the functionality of the Chef Analytics Web UI

• Understand Chef Actions

• Understand audit mode in chef-client 12.1.1

• Understand control recipes and their usage

• Manage Chef on your target instance with the chef-client community cookbook

v 3.0.0

Chef Analytics WebUI

Logon to Chef AnalyticsOpen a browser and go to the public hostname of your analytics ec2 instance.

$ knife node show <firstname-lastname>-analytics-server -a ec2.public_hostname

Logon to Chef Analytics

Chef Analytics WebUI

Chef Actions - Who did what on your Chef Server?

• Provide a read-only view of what happened

• Road to audit and compliance reporting

• Allow administrators to react to events as they happen

• Enable after the fact investigation

• “What happened just before nodes started failing runs?”

• “When did our systems gets patched for Heartbleed?”

Chef Analytics WebUI - Actions

• The run history for all nodes

• Which users made which changes

• Changes made to each node object

• The history of every cookbook (and cookbook version)

• How and where policy settings—roles, environments, and data bags—are applied

Actions

Chef Analytics WebUI - Actions

• Remote hostname

• Request ID

• Chef Server hostname

• Node object information (Previous/Diff/Current)

Actions Details

Chef Analytics WebUI - Search

• Filter Results by Time

Search

Chef Analytics WebUI - Search

• Save - Searches can be saved for later use

Search

Chef Analytics WebUI - Export

• CSV

• JSON

Export

v 3.0.0

chef-client audit-mode

chef-client audit-mode• Released with chef-client version 12.1.1 • Allows the ability to evaluate audit rules • Runs in the following modes:

• disabled (default) - does not run audits • enabled - runs all audits after the chef-client run • audit-only - chef-client run that does not build the resource collection or converge the node

• Can be configured in /etc/chef/client.rb file

v 3.0.0

[2015-03-24T18:30:43+00:00] WARN: Chef-client has been configured to skip converge and run only audits. Audit mode is an experimental feature currently under development. API changes may occur. Use at your own risk. * To enable audit mode after converge, use command line option `--audit-mode enabled` or set `:audit_mode = :enabled` in your config file. * To disable audit mode, use command line option `--audit-mode disabled` or set `:audit_mode = :disabled` in your config file. * To only run audit mode, use command line option `--audit-mode audit-only` or set `:audit_mode = :audit_only` in your config file. Audit mode is disabled by default. Starting Chef Client, version 12.1.1 resolving cookbooks for run list: ["chef-client", "chef-client::config", "chef-client::delete_validation"] Synchronizing Cookbooks: - chef-client - cron - logrotate - windows - chef_handler Compiling Cookbooks... Starting audit phase

Exercise: chef-client --audit-mode audit-only

[chef@hostname ~] sudo chef-client --audit-mode audit-only

v 3.0.0

Update base role

name "base" description "A base role" run_list "recipe[chef-client]", "recipe[chef-client::config]" default_attributes( "chef_client" => { "config" => { "ssl_verify_mode" => ":verify_peer", "audit_mode" => ":enabled", "log_location" => "STDOUT", } } )


http://bit.ly/19tS3vG

http://bit.ly/19tS3vG

v 3.0.0

Updated Role base!

Exercise: Upload role$ knife role from file base.rb

v 3.0.0

Recipe: chef-client::config * template[/etc/logrotate.d/chef-client] action create (up to date) * file[/var/log/chef/client.log] action create (up to date) * template[/etc/chef/client.rb] action create - update content in file /etc/chef/client.rb from 6389f9 to 53658d --- /etc/chef/client.rb 2015-03-24 18:47:39.253084909 +0000 +++ /tmp/chef-rendered-template20150324-30285-1lrucoc 2015-03-24 18:47:58.363085532 +0000 @@ -1,4 +1,6 @@ +audit_mode :enabled chef_server_url "https://52.123.22.235/organizations/analytics-workshop" +log_location STDOUT ssl_verify_mode :verify_peer validation_client_name "analytics-workshop-validator" verify_api_cert true - restore selinux security context * ruby_block[reload_client_config] action create - execute the ruby block reload_client_config

Exercise: re-run chef-client[chef@hostname ~] sudo chef-client

v 3.0.0

Lab Gist1. Update Base role to add audit_mode :enabled /etc/chef/client.rb 2. Upload Base role 3. Re-run chef-client

http://bit.ly/1NC48y1

http://bit.ly/1NC48y1

v 3.0.0

CheckpointAt this point you should have: • ChefDK • A programmers text editor • SSH client • Logged into workshop HipChat channel • Chef Server & Chef Analytics • A node bootstrapped with a base role • audit_mode enabled in /etc/chef/client.rb

v 3.0.0

Controls

Controls• A control is an automated test that is built into a cookbook

• Can be used to test the state of the system for compliance

• Wrapper around ServerSpec testing framework • Can be used to create audit tests around compliance frameworks such as PCI, HIPAA, and Sarbanes-Oxley

http://docs.chef.io/analytics/dsl_recipe.html


Controls Examplecontrol_group "audit name" do control "mysql package" do it "should be installed" do expect(package("mysql")).to be_installed end end end

Control Groups:

• Start with control_group and an end statement




Control Groups:


• Each control_group has a Name




Control Groups:


• Each control group has a Name • control_groups have one or

more control methods




Control Groups:



more control methods • Each control method has Name




Control Groups:



more control methods • Each control method has Name • Each control method can have

one or more it statements that define a specific test to run




Control Groups:



more control methods • Each control method has Name • Each control method can have

one or more it statements that define a specific test to run

• it statements allow for specific tests where expect things .to or .to_not be something



v 3.0.0

Starting audit phase

Audit Mode mysql package should be installed (FAILED - 1)

Failures:

1) Audit Mode mysql package should be installed Failure/Error: expect(package("mysql")).to be_installed expected Package "mysql" to be installed # /var/chef/cache/cookbooks/grantmc/recipes/default.rb:22:in 'block (3 levels) in from_file'

Finished in 0.5745 seconds (files took 0.46481 seconds to load) 1 examples, 1 failures

Failed examples:

rspec /var/chef/cache/cookbooks/grantmc/recipes/default.rb:21 # Audit Mode mysql package should be installed

Example audit output



Controls Matchers• Directory (be_directory, be_directory, be_mounted) • File Matcher (be_executable, be_file, be_grouped_into, contain, be_writable, be_owned_by, etc)

• Service (be_enabled, be_installed, be_running, etc) • Port (be_listening) • Package (be_installed)



v 3.0.0

Compiling Cookbooks... Recipe: code_generator::cookbook * directory[/Users/scottford/analytics-workshop/chef-repo/audit-chef-client] action create - create new directory /Users/scottford/analytics-workshop/chef-repo/audit-chef-client * template[/Users/scottford/analytics-workshop/chef-repo/audit-chef-client/metadata.rb] action create_if_missing - create new file /Users/scottford/analytics-workshop/chef-repo/audit-chef-client/metadata.rb - update content in file /Users/scottford/analytics-workshop/chef-repo/audit-chef-client/metadata.rb from none to 65f7d2 (diff output suppressed by config) * template[/Users/scottford/analytics-workshop/chef-repo/audit-chef-client/README.md] action create_if_missing - create new file /Users/scottford/analytics-workshop/chef-repo/audit-chef-client/README.md - update content in file /Users/scottford/analytics-workshop/chef-repo/audit-chef-client/README.md from none to 694cc1 (diff output suppressed by config)

Exercise: generate a cookbook to audit chef-client

$ chef generate cookbook cookbooks/audit-chef-client

$ cd cookbooks/audit-chef-client

v 3.0.0

Exercise: Create a control recipe

control_group 'Audit Chef Client' do control "Validate chef-client config file" do let(:config_file) { file("/etc/chef/client.rb") } it "should exist with correct permissions" do expect(config_file).to be_mode(644) end it "ssl_verify_mode should be set to :verify_peer" do expect(config_file.content).to match(/ssl_verify_mode :verify_peer/) end end

control 'Check for a validation.pem' do it "The validation.pem should NOT exist" do expect(file("/etc/chef/validation.pem")).to_not be_file end endend

Open in editor: recipes/default.rb

http://bit.ly/1NBDCoA

http://bit.ly/1NBDCoA

Controls Examplecontrol_group 'Audit Chef Client' do

control "Validate chef-client config file" do

let(:config_file) { file("/etc/chef/client.rb") }

it "should exist with correct permissions" do

expect(config_file).to be_file

expect(config_file).to be_mode(644)

end

it "ssl_verify_mode should be set to :verify_peer" do

expect(config_file.content).to match(/ssl_verify_mode :verify_peer/)

end

end

control 'Check for a validation.pem' do

it "The validation.pem should NOT exist" do

expect(file("/etc/chef/validation.pem")).to_not be_file

end

end

end

Audit chef-client

• A control_group • Name ‘Audit Chef Client’ • With two control

statements: • Validate chef-client

config file • Check for a

validation.pem • let method allows you to

define a helper method that can be cached across multiple calls in the same example

v 3.0.0

Uploading audit-chef-client [0.1.0] Uploaded 1 cookbook.

Exercise: Upload audit-chef-client cookbook$ knife cookbook upload audit-chef-client

v 3.0.0

name "base" description "A base role" run_list "recipe[chef-client]", “recipe[chef-client::config]”, “recipe[audit-chef-client]” default_attributes( "chef_client" => { "config" => { "ssl_verify_mode" => ":verify_peer", "audit_mode" => ":enabled", "log_location" => "STDOUT", } } )


http://bit.ly/1NxaQDt

Update base role

http://bit.ly/1NxaQDt

v 3.0.0

Updated Role base!


v 3.0.0

Audit Chef Client Validate chef-client config file should exist with correct permissions ssl_verify_mode should be set to :verify_peer Check for a validation.pem The validation.pem should NOT exist (FAILED - 1)

Failures:

1) Audit Chef Client Check for a validation.pem The validation.pem should NOT exist Failure/Error: expect(file("/etc/chef/validation.pem")).to_not be_file expected `File "/etc/chef/validation.pem".file?` to return false, got true

* directory[/etc/chef/client.d] action create (up to date) * ruby_block[reload_client_config] action nothing (skipped due to action :nothing)


v 3.0.0

Lab Gist1. Create a audit-chef-client cookbook 2. Create a control recipe for auditing chef-client 3. Upload audit-chef-client cookbook 4. Update Base role to add default recipe from audit-chef-client to your node’s run_list 5. Upload Base role to Chef server 6. Re-run chef-client

http://bit.ly/1GaGoi0

http://bit.ly/1GaGoi0

v 3.0.0

CheckpointAt this point you should have: • ChefDK • A programmers text editor • SSH client • Logged into workshop HipChat channel • Chef Server & Chef Analytics • A node bootstrapped with a base role • audit_mode enabled in /etc/chef/client.rb • A control recipe that audits the configuration of chef-client

v 3.0.0

Rules and Notifications

Objectives

• Understand Chef Analytics Notifications and how to implement them

• Understand the Rules in Chef Analytics and how to implement them

• Write Rules that trigger notifications to HipChat

• Install the knife-analytics plugin and use it to create and rules and notifications for Chef Analytics

Notifications

• Adds a language which allows you to express rules on • Run Start • Run End • Resource convergence • Actions

• “When someone not in the ‘siteops’ group modifies the DNS cookbook, alert the siteops team via email to [email protected]”

• “When the /etc/ssh/ssh_config file is modified, raise audit rule 24.1”

mailto:[email protected]

Notification Rulesrule (action) when organization_name = "production" and action = "create" and entity_type = "node" then notify(“smtp”), audit("Rule 3.2 – Node Creation"), log("Fired a rule for org <obj.organization_name>")

Creating Notifications via the WebUI

Creating Notifications via the WebUI1. Open Chef Analytics


2. Click on Notifications


2. Click on Notifications

3. Click the ‘+’ and choose the type of notification you want to set

Creating Notifications via the WebUI1. Give the Notification a

Name

2. Fill in the details

3. Click Save

Creating Rules via the WebUI1. Open Chef Analytics

2. Click on Rules

3. Click the ‘+’ to add a rule

Creating Rules via the WebUI1. Give the Rule a Name


2. Write the rules


2. Write the rules

3. Click Save

v 3.0.0

knife-analytics

v 3.0.0

Successfully installed knife-analytics-0.2.1 1 gem installed

Exercise: Install knife-analytics plugin$ chef gem install knife-analytics

v 3.0.0

# See https://docs.getchef.com/config_rb_knife.html for more information on knife configuration options

current_dir = File.dirname(__FILE__) log_level :info log_location STDOUT node_name "fords" client_key "#{current_dir}/fords.pem" validation_client_name "analytics-workshop-validator" validation_key "#{current_dir}/analytics-workshop-validator.pem" chef_server_url "https://52.123.22.235/organizations/analytics-workshop" analytics_server_url "https://52.123.22.235/organizations/analytics-workshop" syntax_check_cache_path "#{ENV['HOME']}/.chef/syntaxcache" cookbook_path ["#{current_dir}/../cookbooks"]

Exercise: Update knife.rb configuration$ cat .chef/knife.rb

v 3.0.0

WARNING: Certificates from 52.123.22.235 will be fetched and placed in your trusted_cert directory (/Users/scottford/analytics-workshop/chef-repo/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should verify the authenticity of these certificates after downloading.

Adding certificate for 52.123.22.235 in /Users/scottford/analytics-workshop/chef-repo/.chef/trusted_certs/54_68_54_16.crt

Exercise: Fetch Chef Analytics SSL certs$ knife ssl fetch --server-url <CHEF ANALYTICS SERVER>

v 3.0.0

** CHEF ANALYTICS COMMANDS ** knife action list knife action show <id> knife alert list knife alert show <id> knife notification create <notification.json> knife notification list knife notification show <id> knife rule create <rule.json> knife rule list knife rule show <id>

knife-analytics options$ knife analytics help

v 3.0.0

<NO OUTPUT>

Create a local directory for notifications and rules

$ mkdir rules notifications

v 3.0.0

Exercise: Create HipChat Notification for successful rules

{ "org_name": “<YOUR ORG>", "name": "hipchat-success", "modified_by": “<USER NAME>", "notification_type": "hipchat", "delivery_options": { "room": “<HIPCHAT API ID>", "api_token": “<HIPCHAT API TOKEN>", "from": “<firstname-lastname>-analytics", "api_version": "2", "color": "green", "notify": "1" }}

Open in editor: notifications/hipchat_succeed.json

http://bit.ly/1Iz73oe

http://bit.ly/1Iz73oe

v 3.0.0

Exercise: Create HipChat Notification for failed rules

{ "org_name": “<YOUR ORG>", "name": "hipchat-fail", "modified_by": “<USER NAME>", "notification_type": "hipchat", "delivery_options": { "room": “<HIPCHAT API ID>", "api_token": “<HIPCHAT API TOKEN>", "from": “<firstname-lastname>-analytics", "api_version": "2", "color": "red", "notify": "1" }}

Open in editor: notifications/hipchat_fail.json

http://bit.ly/1NjrPcx

http://bit.ly/1NjrPcx

v 3.0.0

<NO OUTPUT>

Create notification with knife$ knife notification create notifications/hipchat_succeed.json

$ knife notification create notifications/hipchat_fail.json

v 3.0.0

Exercise: Create a rule for run_control

{ "name": "Send Control to HipChat", "org_name": “<YOUR ORG>", "modified_by": "<USERNAME>", "rule": "rules \"Run Control\"\n rule on run_control\n when\n status = \"success\"\n then\n notify(\”hipchat-success\", \”<YOUR NAME> had success in run_control \\\"{{ message.name }}\\\" on node {{ message.run.node_name }}\")\n end\n\n rule on run_control\n when\n status = \"failure\"\n then\n notify(\”hipchat-failure\", \"<YOUR NAME> has a FAILURE on run_control \\\"{{ message.name }}\\\" on node {{ message.run.node_name }}\")\n end\nend", "with": { "priority": 0 }, "active": true}

Open in editor: rules/send_hipchat.json

http://bit.ly/1CFNom2

http://bit.ly/1CFNom2

v 3.0.0

<NO OUTPUT>

Create notification with knife$ knife rule create rules/send_hipchat.json

v 3.0.0

Audit Chef Client Validate chef-client config file should exist with correct permissions ssl_verify_mode should be set to :verify_peer Check for a validation.pem The validation.pem should NOT exist (FAILED - 1)

Failures:

1) Audit Chef Client Check for a validation.pem The validation.pem should NOT exist Failure/Error: expect(file("/etc/chef/validation.pem")).to_not be_file expected `File "/etc/chef/validation.pem".file?` to return false, got true

* directory[/etc/chef/client.d] action create (up to date) * ruby_block[reload_client_config] action nothing (skipped due to action :nothing)


Login To HipChat1. Give the Notification a

Name


3. Click Save

v 3.0.0

Update base role

name "base" description "A base role" run_list "recipe[chef-client]", “recipe[chef-client::config]”, “recipe[chef-client::delete_validation]”, “recipe[audit-chef-client]” default_attributes( "chef_client" => { "config" => { "ssl_verify_mode" => ":verify_peer", "audit_mode" => ":enabled", "log_location" => "STDOUT", } } )


http://bit.ly/1CrlcSo

http://bit.ly/1CrlcSo

v 3.0.0

Updated Role base!


v 3.0.0

Recipe: chef-client::delete_validation * file[/etc/chef/validation.pem] action delete - delete file /etc/chef/validation.pem Starting audit phase

Audit Chef Client Validate chef-client config file should exist with correct permissions ssl_verify_mode should be set to :verify_peer Check for a validation.pem The validation.pem should NOT exist

Finished in 0.46837 seconds (files took 0.60923 seconds to load) 3 examples, 0 failures Auditing complete

Running handlers: Running handlers complete Chef Client finished, 1/16 resources updated in 12.113568488 seconds 3/3 Audits succeeded


Login To HipChat1. Give the Notification a

Name


3. Click Save

v 3.0.0

Lab Gist1. Install knife-analytics plugin 2. Create a rules and notifications directory in chef-repo 3. Create a notification configuration with HipChat for successful and failed audits 4. Create a rule to send to HipChat upon successful and failed chef-client runs

http://bit.ly/1Ms6T81

http://bit.ly/1Ms6T81

v 3.0.0

CheckpointAt this point you should have: • ChefDK • A programmers text editor • SSH client • Logged into workshop HipChat channel • Chef Server & Chef Analytics • A node bootstrapped with a base role • audit_mode enabled in /etc/chef/client.rb • A control recipe that audits the configuration of chef-client • A notification configured to integrate with HipChat • A rule configured to send notifications to HipChat based on successful and failed audit rules

v 3.0.0

Q&A Time

Further Resources

• http://docs.chef.io/analytics/

• http://serverspec.org/

• https://github.com/opscode/chef-provisioning

• https://github.com/opscode/chef-provisioning-aws

http://serverspec.org/

https://github.com/opscode/chef-provisioning

https://github.com/opscode/chef-provisioning-aws

Documents

chefconf 2015 intro to analytics - Viewdeck - Home · 2016-03-22 · Web Servers Application Servers Database. Exponential Increase in Size Leads to ... • Success/Failure status