James Andrew Vaughn (Andy) @MindTouch

Tear It Down, Build It Back Up: Empowering Developers with

Amazon CloudFormation

James Andrew Vaughn (Andy)

• Software Architect at MindTouch• @modethirteen on Twitter & GitHub• Interests

• Software Build and Testing Automation• Frontend Web Performance• Web Components & Polymer• SSO and Identity Management

@modethirteen

Agenda• What is Amazon CloudFormation? Why use it?

• Managing your release testing and production infrastructure code

• Give developers the power (`cause knowledge is power!)

@modethirteen

Why manage infrastructureas code?

@modethirteen

@modethirteen

All of our customers host their brand on our common, hosted infrastructure.

One mistake and all customer brands look bad #yousuck

@modethirteen

Before CloudFormation• Infrastructure had grown organically over years

• Hand rolled scripts with boto.py to create different EC2 instance types, and manual Puppet runs to configure them

• Non EC2 AWS Resources managed by hand

• No infrastructure in different zones or fast, programatic disaster recovery for entire infrastructure

• Developers were ignorant of production infrastructure

@modethirteen

Weekly releases must be simple, repeatable, non

events

@modethirteen

Developers cannot be isolated from the infrastructure where their code will ultimately run

@modethirteen

Code gives context to problems solved and provides audit trail

for infrastructure design

@modethirteen

Infrastructure code and server configuration code is versioned

with application code

@modethirteen

CloudFormation: Define creation of AWS resources (EC2 as well as Security Groups, SQS, RDS, etc)

Puppet, Chef, SaltStack, Ansible: Define actions that occur within EC2 instances once they’ve been provisioned

@modethirteen

CloudFormation vs Terraform• Access to nearly every AWS

resource. Better support for VPC, Security Groups, IAM, Cloudfront, SQS

• Stable and mature

• JSON infrastructure templates can be generated by Troposphere (with Python logic)

• Vendor neutrality: AWS, OpenStack, Heroku, etc

• Can execute infrastructure plans as a dry run

• DSL for generating infrastructure templates (HCL)

• If one resource fails to build, subsequent rebuild will only build tainted resource and those dependent on it

• Open source so AWS API coverage can be improved by community

Google Docs: Terraform AWS Coverage

@modethirteen

CloudFormation Stacks

Main Stack

Sub Stacks

A stack is a collection of AWS resources that can be configured

@modethirteen

App Server PoolStack

DatabaseStack

ElasticSearchStack

App Server PoolStack

Main Stack

@modethirteen

CloudFormation StacksResources are things that can be queried, configured in the AWS API (including

CloudFormation sub stacks). Examples: Listing S3 buckets, Adding Route 53 DNS entries, Taking DB snapshots

@modethirteen

DatabaseStack

ElasticSearchStack

App Server PoolStack

Main Stack

• AutoScaling::AutoScalingGroup• AutoScaling::LaunchConfiguration• IAM::InstanceProfile• IAM::User

• AutoScaling::AutoScalingGroup• AutoScaling::LaunchConfiguration• CloudFormation::WaitCondition• IAM::InstanceProfile• IAM::User

• RDS::DBInstance• IAM::InstanceProfile• IAM::User

@modethirteen

Custom Resources

• CloudFormation::CustomResource

• Sends custom HTTP message (Service Token) to any of your endpoints, and continues stack execution after response

• AWS SNS

• AWS Lambda

• Node.JS

• Your choice!

@modethirteen

CloudFormation StacksStack parameters come from API input, version controlled JSON

templates, or from the output of other stacks

@modethirteen

• MySQL Storage Engine

App Server PoolStack

DatabaseStack

ElasticSearchStack

App Server PoolStack

Main Stack

• ElasticSearch Version• App Server Pool EC2 Group Name

• ElasticSearch EC2 Group Name• RDS MySQL IP & Port

@modethirteen

CloudFormation StacksParameters of stack can be outputted to dependent stacks.

Example: IP’s, Security Policies, Custom Values, etc.

@modethirteen

Template: {…}

App Server PoolStack

DatabaseStack

ElasticSearchStack

App Server PoolStack

Main Stack

• MySQL Storage Engine

• ElasticSearch EC2 Group Name• RDS MySQL IP & Port

• ElasticSearch Version• App Server Pool EC2 Group Name

Template: {…}

@modethirteen

Stack Policy: Stack Update Resource Access Control

@modethirteen

Deploying a Stack

@modethirteen

Troposphere

@modethirteen

Puppet / Chef / SaltStack / Ansible

• Stack includes an EC2 Instance or AutoScaling Group Resource

• Resource includes a “UserData” metadata section, for bootstrapping an instance or group of instances

• Include data that cloud-init uses to install instance configuration tool of choice

• curl http://169.254.169.254/latest/user-data

• Example:

• cloud-init installs puppet from UserData commands

• cloud-init runs puppet (configures instance and installs cfn-signal)

• cfn-signal notifies CloudFormation that puppet was success or failure

@modethirteen

Execute Deployment

@modethirteen

Lessons Learned• Goal was to put entire existing AWS infrastructure into

CloudFormation, no immediate value was attained

• Difficult getting buy in for incremental improvements to infrastructure management

• Existing resources cannot be migrated to CloudFormation

• Know the caveats of deleting AWS Resources, they can fail a stack tear down

• AWS Resources missing from CloudFormation API can be mitigated with Custom Resources

• Must understand what a resource does when it updates

@modethirteen

Send in the Developers

@modethirteen

Approach #1 : Build your own web console for launching test

and dev stacks

@modethirteen

Approach #2 : Every developer has their own AWS account billed to main AWS account

@modethirteen

Approach #3 : One developer AWS account billed to main

account

@modethirteen

The Teams

• Are developer teams responsible for their own container / infrastructure templates, are operators part of these teams

• Are developers just as responsible for troubleshooting when infrastructure goes down

• What are operator obligations to developers

• What are developer obligations to operations

@modethirteen

TL;DR• Your product is application code, data, services, and

servers

• CloudFormation deploys your product to production

• CloudFormation deploys your product for development and testing

• Your developers can make better decisions

• Your operators can make better decisions

• Your customers / users are happy

The End. Q?