Text

CloudFormation at 99designs@lox / @99designs

How we use CFN

Each product team has their own formations in their product git repository.

Some teams use it to provision Docker environments, some use it to provision complete production servers

Increasingly, more teams are using it to provision all of the AWS resources that make up a web stack, and document that process in an OPS readme.

Infrastructure as code, versioned templates side-by-side with code

PrinciplesBuild a hierarchy of loosely-coupled stacks

Group resources that are strongly coupled

Template inputs for things that are dependencies (databases, AMIs, external ELBs, Route53 zones)

Template outputs should make up inputs to other stacks

Design based around things that change at the same time, e.g. split ELBs and auto-scaling app server groups

Principles

Keep it simple

Automate the building of pieces that can be assembled by humans

Invest the most automation in parts of the system that change the most frequently.

JSON is hard to read, keep templates short

Layers on Layers

Lots of tools for managing, transforming and generating CFN templates

https://github.com/cloudtools/troposphere

https://github.com/cotdsa/cumulus

At present we don’t use any of these, but might in future

Best Practice:

Use WaitHandle, WaitCondition and DependsOn

WaitHandles wait for resources to finish provisioning

Enforce ordering with DependsOn

Errors in UserData should signal the WaitHandle immediately. Failing fast means quicker iterating on templates.

"WaitHandle" : { "Type" : "AWS::CloudFormation::WaitConditionHandle"},"WaitCondition" : { "Type" : "AWS::CloudFormation::WaitCondition", "DependsOn" : "AppServers", "Properties" : { "Handle" : { "Ref" : "WaitHandle" }, "Timeout" : "300", "Count" : { "Ref" : "NumberOfAppServers" } }}

"# signal to the waithandle\n","cfn-signal -e $? -r \"Setup complete\" '", { "Ref" : "WaitHandle" }, "'\n"

Best Practice:

Use aws-cfn-bootstrap to keep userdata short

Userdata is the hardest part to read, so keep it short

A single call to cfn-init means consolidated error handling

Configure resources declaratively via Metadata

"UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ "#!/bin/bash -x\n", "exec &>/home/ubuntu/boot.log\n", "tail -F /var/log/cfn-init.log &\n",

"# install cfn-bootstrap\n", "apt-get -y install python-setuptools\n", "easy_install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n", "cfn-init -v ", " --region ", { "Ref" : "AWS::Region" }, " --resource LaunchConfig", " --stack ", { "Ref" : "AWS::StackName" },"\n",

"# signal to the waithandle\n", "cfn-signal -e $? -r \"Setup complete\" '", { "Ref" : "WaitHandle" }, "'\n"]]}

Best Practice:

Use AWS metadata for init and authentication

Download private setup files from S3

Install packages via yum/apt

Execute ad-hoc commands

Keeps error handling to a minimum, DRY code

"LaunchConfig" : { "Type" : "AWS::AutoScaling::LaunchConfiguration", "Metadata" : { "AWS::CloudFormation::Authentication" : { "S3AccessCreds" : { "type" : "S3", "roleName" : { "Ref" : "IAMRole" }, "buckets" : [ "my-secrets" ] }}, "AWS::CloudFormation::Init" : { "config" : { "sources" : { "/root/provision" : { "Fn::Join" : ["", ["https://s3.amazonaws.com/my-secrets/", { "Ref" : "ScriptsTarball" } ]] }}}}}}

Best Practice:

Security groups for shared access, instance roles for everything else

For inter-template security groups, use named security groups

Using instance roles provides token based credentials to the instance, compatible with aws cli tools

Avoid leaking credentials via templates

"Parameters" : { "RDSAccessSecurityGroup" : { "Description" : "Security group to connect to RDS via", "Type" : "String", "Default" : "rds-access" }, "BeanstalkdAccessSecurityGroup" : { "Description" : "Security group to connect to beanstalkd via", "Type" : "String", "Default" : "beanstalkd-access" }, "RedisAccessSecurityGroup" : { "Description" : "Security group to connect to elasticache/redis", "Type" : "String", "Default" : redis-access" },}

Best Practice:

Bootstrap instances from scripts in private s3 buckets

Rather than checking out a git repo, when you create a stack, link it to a tarball of scripts

Easy to iterate, consistent instance launches for future auto-scale events

"LaunchConfig" : { "Type" : "AWS::AutoScaling::LaunchConfiguration", "Metadata" : { "AWS::CloudFormation::Authentication" : { "S3AccessCreds" : { "type" : "S3", "roleName" : { "Ref" : "IAMRole" }, "buckets" : [ "secret-stuff" ] } }, "AWS::CloudFormation::Init" : { "config" : { "sources" : { "/root/provision" : { "Fn::Join" : ["", ["https://s3.amazonaws.com/secret-stuff/", { "Ref" : "ProvisionSlug" } ]] } }, }}}}

Best Practice:

Automate monitoring

CloudWatch is great, use it

Automate alarms, don’t leave them as an afterthough

Good clouds are ones with monitoring!

"SNSTopic" : { "Type" : "AWS::SNS::Topic", "Properties" : { "Subscription" : [{ "Endpoint" : { "Ref": "AlertEmailAddress" }, "Protocol" : "email" }] }},"CPUAlarmDB" : { "Type" : "AWS::CloudWatch::Alarm", "Properties" : { "AlarmDescription": { "Fn::Join" : ["", [{ "Ref" : "RDSInstance" }, "DB CPU Utilization"]]}, "MetricName": "CPUUtilization", ... "AlarmActions": [ { "Ref": "SNSTopic" } ], "Dimensions": [{ "Name": "DBInstanceIdentifier", "Value": { "Ref": "RDSInstance" } }], "ComparisonOperator": "GreaterThanOrEqualToThreshold" }},

Best Practice:

NoEcho property for passwords

Prevent information leak for sensitive parameters

"DockerPassword": { "NoEcho": "true", "Description" : “Docker index password", "Type": "String", "MinLength": "1", "MaxLength": "41" }

Questions we have

Reuse beyond copy/paste?

Elegant AMI selection using Fn::select?

Update stacks, or destroy/recreate?

CFN for Route53?

Future plans

CloudWatch streaming logs, see http://blogs.aws.amazon.com/application-management/post/TxPYD8JT4CB5UY/View-CloudFormation-Logs-in-the-Console

Nested templates to manage dependent templates

Elastic Beanstalk

Questions?

Resources

https://gist.github.com/lox/66a78542b2c14a3f773d

http://blogs.aws.amazon.com/application-management/blog/tag/CloudFormation

http://harish11g.blogspot.com.au/2014/08/amazon-cloudformation-templates-automation-Amazon-CFT-AWS-top-best-practices-tips.html

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-authentication.html

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html