Embed Size (px)
Citation preview
© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Infrastructure as code Thomas Metschke, AWS
May 15, 2014
Management services
Convenience Control
AWS Elastic Beanstalk as well as AWS OpsWorks can be controlled via AWS CloudFormation.
AWS CloudFormation integration
AWS CloudFormation integration
Stack
Layer / Services
Instances
Agent § Installed on every instance § Can execute different commands on an instance § Sends keep alive messages for auto healing § And 14 host level metrics every minute
Lifecycle events
Launch first instance
Setup triggers configure event
Deploy the static App
Add a database instance
Reconfigure Stack
Deploy and migrate database
Add more instances
Configure Stack
Execute recipes – any time
Stop instance
Configure Stack
§ Open Source § Configuration as code
Documented Versioned Testable Reusable
§ Cookbooks & recipes
Chef
Basic cookbook folder structure ▾ apache2 ▸ attributes ▸ recipes ▸ templates – CHANGELOG.md – metadata.rb – README.md
§ Cookbook description § Maintainer contact § License § Version § Dependencies § Conflicting cookbooks § Supported platforms
Cookbook metadata.rb ▾ apache2 ▸ attributes ▸ recipes ▸ templates – CHANGELOG.md – metadata.rb – README.md
Cookbook metadata.rb name 'apache2' !maintainer 'Darth Vader' !license 'Death Star Commons 2.0' !description 'Installs and configures apache2' !version '0.0.1' !recipe 'apache2', 'Apache setup' !!supports 'amazon' !supports 'ubuntu' !
metadata.rb
Environment data; the node object
Ohai Attributes Node Object
§ Runs on the instance
§ Detects host attributes
§ Adds them to the node object
{ ! "platform": "amazon", ! "platform_version": "2013.09", ! "os": "linux", ! "counters": { ! "network": { ! "interfaces": { ! "eth0": { ! ... ! "kernel": { ! "os": "GNU/Linux", ! "name": "Linux", ! "modules": { !... !} !
Ohai
§ Define variations or properties
§ Everything that may be subject to change over time or due to use case
Cookbook – attributes ▾ apache2 ▸ attributes
default.rb ▸ recipes ▸ templates – CHANGELOG.md – metadata.rb – README.md
Attributes default.rb case node[:platform] !when 'redhat','centos','fedora','amazon' ! default[:apache][:dir] = '/etc/httpd' ! default[:apache][:user] = 'apache' ! default[:apache][:group] = 'apache' ! default[:apache][:document_root] = '/var/www/html' !when 'debian','ubuntu' ! default[:apache][:dir] = '/etc/apache2' ! default[:apache][:user] = 'www-data' ! default[:apache][:group] = 'www-data' ! default[:apache][:document_root] = '/var/www' !else ! raise 'Unknown platform' !end !
apache2 / attributes / default.rb
Attributes default.rb case node[:platform] !when 'redhat','centos','fedora','amazon' ! default[:apache][:dir] = '/etc/httpd' ! default[:apache][:user] = 'apache' ! default[:apache][:group] = 'apache' ! default[:apache][:document_root] = '/var/www/html' !when 'debian','ubuntu' ! default[:apache][:dir] = '/etc/apache2' ! default[:apache][:user] = 'www-data' ! default[:apache][:group] = 'www-data' ! default[:apache][:document_root] = '/var/www' !else ! raise 'Unknown platform' !end !
apache2 / attributes / default.rb
§ Scripts that get executed on the instances
§ Referenced as apache2 !apache2::setup !based on metadata.rb
Cookbook – recipes ▾ apache2 ▸ attributes ▾ recipes
default.rb setup.rb configure.rb service.rb
▸ templates
§ setup.rb § Idea is to create all
necessary steps to install Apache on a vanilla OS
Cookbook – setup.rb recipe ▾ apache2 ▸ attributes ▾ recipes
default.rb setup.rb configure.rb service.rb
▸ templates
Recipe setup.rb package 'apache2'!
apache2 / recipes / setup.rb
package !template !service !!cron!log !group / user !
link !directory / remote_directory!file / remote_file!!ruby / perl / python / bash !execute / ruby_block!deploy !
Chef resources
Resources describe what to do or define the desired state Resources have § attributes § actions
cron "daily_report" do ! minute "0" ! hour "0" ! day "*" ! command "/daily_report" ! action :create !end !
Chef resources
Recipe setup.rb package 'apache2' do !"action :install !
end !
apache2 / recipes / setup.rb
Recipe setup.rb package 'apache2' do ! case node[:platform] ! when 'centos','redhat','fedora','amazon' ! package_name 'httpd' ! when 'debian','ubuntu' ! package_name 'apache2' ! end ! action :install !end !
apache2 / recipes / setup.rb
Recipe setup.rb package 'apache2' do ! #case node[:platform] ! ... !end !!include_recipe 'apache2::configure' !
apache2 / recipes / setup.rb
§ Create or change Apache configuration
§ Included by the setup recipe
Cookbook – configure.rb recipe ▾ apache2 ▸ attributes ▾ recipes
default.rb setup.rb configure.rb service.rb
▸ templates
Recipe configure.rb template 'apache2.conf' do ! source 'apache2.conf.erb' !end "!
apache2 / recipes / configure.rb
Recipe configure.rb template 'apache2.conf' do ! case node[:platform] ! when 'centos','redhat','fedora','amazon' ! path "#{node[:apache][:dir]}/conf/httpd.conf" ! when 'debian','ubuntu' ! path "#{node[:apache][:dir]}/apache2.conf" ! end ! source 'apache2.conf.erb' !end "!
apache2 / recipes / configure.rb
§ Embedded Ruby (ERB) § Variable substitution
and flow control (conditionals or loops)
§ Used for parameterized file generation (configurations, scripts)
Cookbook – templates ▾ apache2 ▸ attributes ▸ recipes ▾ templates ▾ default
apache2.conf.erb
§ Can be fundamentally different per platform and platform version
§ Rendering is triggered via calling the template resource
Cookbook – templates per platform ▾ apache2 ▸ attributes ▸ recipes ▾ templates ▾ default
apache2.conf.erb ▾ centos-5.7
apache2.conf.erb
Template apache2.conf.erb ServerName 127.0.0.1 !Listen *:80 !DocumentRoot "<%= node[:apache][:document_root] %>" !!User <%= node[:apache][:user] %> !Group <%= node[:apache][:user] %> !ServerRoot "<%= node[:apache][:dir] %>" !!... !
apache2 / templates / default / apache2.conf.erb
§ Apache service definition
§ How does start, stop, restart, reload, enable, disable of the service work
Cookbook ▾ apache2 ▸ attributes ▾ recipes
default.rb setup.rb configure.rb service.rb
▸ templates
Recipe service.rb service 'apache2' do ! #need to add cases for different platforms ! service_name 'httpd' ! restart_command '/sbin/service httpd restart' ! reload_command '/sbin/service httpd reload' !! supports [:restart, :reload, :status] ! action [:nothing] !end !
apache2 / recipes / service.rb
Recipe configure.rb template 'apache2.conf' do ! case node[:platform] ! ... ! end ! source 'apache2.conf.erb' ! notifies :reload, 'service[apache2]' !end "!
apache2 / recipes / configure.rb
§ Provide the cookbook source for the Stack (git, svn, http, S3)
§ Assign the recipes to the lifecycle events
Use the cookbook in OpsWorks
Setup event Configure event
1. Base recipes are run 2. Apache configure recipe is
run, without any change on the instance
OpsWorks events breakdown
1. Base recipes are run 2. Apache setup recipes
installs the package 3. Apache configure recipe
writes configuration 4. Notify Apache service
recipe to start Apache
¢
Community cookbooks http://community.opscode.com/search § Close to 1.500 cookbooks § Ratings and descriptions § Versions history § Will be reworked soon
Community cookbooks https://github.com/search?q=cookbook § Search returns over 10.000 repositories § Not all of them are Chef cookbooks though § Not all of the cookbooks are perfect, check issues,
stars, watchers, forks, number of contributors, …
Berkshelf support § Version 2 and 3 § Be aware of different syntax between versions § Manage cookbook dependencies § Put your Berksfile in your root folder
Berkshelf support source "https://api.berkshelf.com" !#metadata !!#cookbook {name}, {version_constraint}, {options} !cookbook "mysql" !cookbook "nginx", "~> 2.6" !cookbook "mysql", ! git: "https://github.com/mystuff/mysql.git", ! branch: "master-master_v2" !
Precedence with Chef 11.10 1. Ohai 2. Deployment JSON 3. Custom JSON 4. Your Cookbooks 5. Berkshelf Cookbooks 6. Built-in Cookbooks
Precedence of attributes 1. Ohai 2. Deployment JSON 3. Custom JSON 4. Your Cookbooks 5. Berkshelf Cookbooks 6. Built-in Cookbooks
Search § Includes all attributes § Adds Ohai data from the instance itself § Your are not able to search for Ohai data
of other instances in the Stack
Recipe hosts.rb template '/etc/hosts' do ! source "hosts.erb" ! mode "0644" ! variables( ! :nodes => search(:node, "name:*") ! ) !end !
Template hosts.erb ... !!<% @nodes.each do |node| -%> <%= ! node.attributes.private_ip %> <%= ! node.hostname %> <%= ! node.name %> !<% end -%> !!... !
Example result hosts ... !!10.95.180.186 cupcake cupcake.localdomain!10.92.124.141 croissant croissant.localdomain!10.143.21.126 french-toast french-toast.localdomain!10.195.180.64 cheesecake cheesecake.localdomain!10.35.203.124 pie pie.localdomain!!... !
Another example # Retrieve the master node !master = search(:node, "role:layer_shortname").first !!# Show the private IP of it in the Chef log !Chef::Log.info("Private IP: #{master[:private_ip]}") !!!
Logging Chef::Log.info('something useful') !# [timestamp] INFO: something useful !!Chef::Log.error('something hit the fan') !# [timestamp] ERROR: something hit the fan !!Chef::Log.info("Node object: #{node.to_json}") !# [timestamp] INFO: Node object: {...} !!
Node object snippets { "normal": { ! "opsworks": {"activity": "setup", ... }, ... } ! "automatic": { ! "languages": { ... }, ! "ec2": { "ami_id": "", "iam": {}, ... }, ! "cpu": { ... }, ! "memory": { ... }, ! "filesystem": { ... }, ! "uptime_seconds" => 250217 }, ... } !
Foodcritic § A linting tool for your Chef cookbooks § Comes with a set of rules § Choose which to use and add your own § Run with foodcritic [cookbook_path] !
Example rule FC007 § Ensures recipe dependencies are reflected in
cookbook metadata files § You use include_recipe "apache2::default"
in a recipe § But you miss depends "apache2"
in the metadata.rb!
Kitchen CI Run infrastructure tests with different § Drivers (OpsWorks, EC2, Vagrant, Docker, …) § Platforms (like Ubuntu, Amazon Linux, …) § Suites (what to test)
§ Test your logic not Chef base functionality!
Kitchen CI .kitchen.yml driver: ! name: vagrant !!provisioner: ! name: chef_zero!!platforms: ! - name: ubuntu-13.04 ! - name: centos-6.5 !
Kitchen CI .kitchen.yml suites: ! - name: client ! run_list: ! - recipe[postgresql::client] ! - name: server ! run_list: ! - recipe[postgresql::server] !
Run Kitchen CI # destroy, create, converge, setup, verify, destroy !kitchen test !!kitchen verify [NAME|REGEXP|all]"# run all 'tests' !kitchen verify ubuntu " " " "# only ubuntus!kitchen verify client " " " "# only client tests !kitchen verify -c 2 " " " " "# concurrency !!# Talk by Nichol Fletcher http://youtu.be/YzlCHAbJ7KM !!
§ Explains vagrant setup and configuration
§ Follows along user OpsWorks guide
§ Development and deployment covered
AWS OpsWorks with Vagrant
http://pixelcog.com/blog/2014/virtualizing-aws-opsworks-with-vagrant/
Develop on a live instance opsworks-agent-cli commands that can help you develop your cookbooks !
list_commands list the commands get_json show the JSON used run_command re-run a command show_log show the current or old log
Agent CLI – list what happened # list event history (up to 10 events) ![root@pie thomas]# opsworks-agent-cli list !2014-05-06T10:26:58 setup !2014-05-06T10:55:13 configure !2014-05-06T14:59:05 execute_recipes!2014-05-07T18:25:53 " " "deploy !2014-05-07T18:34:11 configure !2014-05-08T10:05:23 configure !2014-05-08T14:43:43 configure !
Make a change to your cookbooks # list current agent folder and cookbooks ![root@pie thomas]# ls /opt/aws/opsworks/current!!cookbooks " " " " "# OpsWorks cookbooks!berkshelf-cookbooks " "# Cookbooks from Bershelf !site-cookbooks" " " "# Your cookbooks!merged-cookbooks " " "# Resulting cookbooks for run !... !
Agent CLI – re-run an event # run last setup event (changed cookbooks, same JSON) ![root@pie thomas]# opsworks-agent-cli run setup !!# or select specific date ![root@pie thomas]# opsworks-agent-cli run timestamp!# [2014-05-12 13:06:41] INFO ! [opsworks-agent(17817)]: About to re-run 'setup' ! from 2014-05-08T14:43:43 ! ... !
Agent CLI – show the current/last log # show the latest event ![root@pie thomas]# opsworks-agent-cli show !!# show the last setup event![root@pie thomas]# opsworks-agent-cli show setup !!# show a specific event based on a timestamp![root@pie thomas]# opsworks-agent-cli show timestamp !!
Agent CLI – get the JSON to debug # show JSON events (latest or by type/timestamp) ![root@pie thomas]# opsworks-agent-cli get !{ ! "ssh_users": { ! "2066": { ! "name": "thomas", ! "public_key": "ssh-rsa AAAAB3NzaC1...", ! "sudoer": true !... !
Interesting topics I didn't touch today § Github hooks § Continuous Integration § Docker or any other technology XYZ § Use (encrypted) databags in AWS OpsWorks § Silver AMI to reduce boot times § Chef profiler – chef-handler-profiler
Keep in touch
@AWSOpsWorks on twitter blogs.aws.amazon.com/application-management
Send us feedback
Send us feedback
