Continuous Deployment @ AWS Re:Invent

  • View
    1.395

  • Download
    4

Embed Size (px)

DESCRIPTION

Leo Z and I gave a talk this year at AWS Re:Invent on Continuous Integration at OFA, Mozilla Foundation, and other companies.

Text of Continuous Deployment @ AWS Re:Invent

  • 1. Continuous Integration and Deployment Best Practices on AWS - ARC307 Leo Zhadanovsky, Senior Solutions Architect, AWS @leozh JP Schneider, DevOps / Internet Jedi, Mozilla Foundation @jdotp November 13th, 2013 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • 2. Who Am I? I work for AWS I worked for the DNC 2009-2012 I was embedded in the DevOps Team @ OFA AWS does not endorse political candidates
  • 3. Who Am I? JP, DevOps for Mozilla Foundation Previous gig DevOps at OFA 2012 Before that, Ops at Threadless @jdotp Mozilla Foundation does endorse animated cats
  • 4. CONTINUOUS INTEGRATION
  • 5. What is Continuous Integration? Changes to code automatically deployed to mainline branch After passing unit and mock tests Makes changes to code and deployments iterative, not monolithic Bugs are detected quickly Allows rapid development Helps automate deployments
  • 6. DEVELOPER
  • 7. SOURCE CODE REPOSITORY
  • 8. SOURCE CODE REPOSITORY PROJECT MANAGEMENT SERVER
  • 9. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER PROJECT MANAGEMENT SERVER
  • 10. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER PROJECT MANAGEMENT SERVER PICK TASKS
  • 11. SOURCE CODE REPOSITORY SUBMIT CODE CONTINUOUS INTEGRATION SERVER PROJECT MANAGEMENT SERVER
  • 12. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER SCHEDULE BUILD PROJECT MANAGEMENT SERVER
  • 13. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER RECURRENT BUILDS PROJECT MANAGEMENT SERVER
  • 14. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER CODE FETCH PROJECT MANAGEMENT SERVER
  • 15. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER CODE QUALITY TESTS PROJECT MANAGEMENT SERVER TEST RESULTS
  • 16. SOURCE CODE REPOSITORY CONTINUOUS INTEGRATION SERVER BUILD OUTPUT PROJECT MANAGEMENT SERVER
  • 17. SOURCE CODE REPOSITORY DOCS CONTINUOUS INTEGRATION SERVER BINARIES & PACKAGES PROJECT MANAGEMENT SERVER DEV FACING NOTIFICATIONS
  • 18. SOURCE CODE REPOSITORY BUILDS CONTINUOUS INTEGRATION SERVER DNS PROJECT MANAGEMENT SERVER
  • 19. PAIN POINTS UNIT TESTS INCOMPLETE MOCK TESTS MAINTENANCE EXPENSIVE TEST ENVIRONMENT TEST ENVIRONMENT PRODUCTION DEPLOYMENT CYCLES
  • 20. ON-DEMAND ELASTIC PAY AS YOU GO
  • 21. = PROGRAMMABLE PLATFORM
  • 22. IF YOU CAN PROGRAM IT YOU CAN AUTOMATE IT
  • 23. A lot of options Configuration Management Systems Puppet Chef Saltstack Deployment Frameworks AWS Elastic Beanstalk AWS OpsWorks Ansible Fabric Capistrano Infrastructure Management AWS CloudFormation
  • 24. Bootstrapping Bake an AMI Time consuming configuration (startup time) Static configurations (less change management) Configure dynamically
  • 25. Bootstrapping Bake an AMI Configure dynamically Continuous deployment (latest code) Environment specific (devtest-prod)
  • 26. Obama for America awsofa.info
  • 27. Web-Scale Applications
  • 28. 500k+ IOPS DB Systems
  • 29. Services API
  • 30. Typical Charts
  • 31. How?
  • 32. Ingredients Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB
  • 33. Infrastructure, Configuration Management & Monitoring Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS OpsView Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB
  • 34. Configuration Management: Puppet In mid-2011, we looked at options for configuration management and chose Puppet We needed to make it scale, and to get it to work with stateless, horizontally scalable infrastructure How did we do this?
  • 35. Bootstrapping Puppet with CloudInit CloudInit is built into Ubuntu and Amazon Linux Allows you to pass bootstrap parameters in Amazon EC2 user-data field, in YAML format
  • 36. Bootstrapping Puppet with CloudInit Dont store creds in puppet manifests, store them in private Amazon S3 buckets Either pass Amazon S3 creds through CloudInit: Even better avoid this by using AWS Identity and Access Management (IAM) roles and AWS Unified CLIs S3 client
  • 37. Bootstrapping Puppet with CloudInit Built-in puppet support Use certname with %i for instance id to name the node Puppetmaster must have auto sign turned on Use security groups and/or NACLs for network-level security In nodes.pp, use regex to match node names
  • 38. Puppet Tips Use a base class to define your standard install
  • 39. Puppet Tips Use runstages Dont store credentials in puppet, store them in private Amazon S3 buckets Use AWS IAM to secure the credentials bucket/folders within that bucket
  • 40. Puppet Tips Use puppet only for configuration files and what makes your apps unique For undifferentiated parts of apps, use Amazon S3 backed RPM/Debian repositories Can be either public or private repos, depending on your needs Amazon S3 Private RPM Repos: http://git.io/YAcsbg Amazon S3 Private Debian Repos: http://git.io/ecCjWQ
  • 41. Puppet Tips By using packages for applications deploys, you can set ensure => latest, and just bump the package in the repo to update Log everything with rsyslog/graylog/loggly/NewRelic/splunk
  • 42. Scaling the Puppet Masters Use an Auto Scaling group for puppet masters Min size => 2, use multiple Availability Zones Either have them build themselves off of existing puppet masters in the group or off packages stored in Amazon S3 and bootstrapped through user-data Auto-sign must be on
  • 43. One thing that