Upload
noah-zoschke
View
1.160
Download
0
Embed Size (px)
Citation preview
╔══════════════════════════════════════════╗ ║ CHOOSE YOUR OWN AWS ADVENTURE ║ ║ ║ ║ Noah Zoschke ║ ║ [email protected] ║ ║ @nzoschke ║ ║ ║ ║ 1/28/2016 ║ ╚══════════════════════════════════════════╝
CONVOX
CONVOX DEVOPS TOOLS AND SERVICES
Eliminate complex and boring parts of architecting a system on AWS
Automate the hard parts of operating software on AWS
Integrate the best pieces from AWS and Docker ecosystem
Open source everything to share code and expertise
Support and maintain for app and infrastructure lifecycle
CONVOX OPEN SOURCE PAAS ⟷ IAAS
Racks ⟷ ASG, CF, Dynamo, EC2, ECS, IAM, VPC
Apps ⟷ CF, ECS, ELB
Scale ⟷ ASG, CF, ECS
Environments ⟷ KMS, S3
Builds ⟷ S3, ECR
Logs ⟷ CloudWatch Logs, Kinesis, Lambda
Metrics ⟷ CloudWatch Metrics, Elasticsearch
Notifications ⟷ SNS
CHOOSE YOUR OWN ADVENTURE
BUILD AND PACKAGING DECISION TREE
Build AMIs
What OS? ubuntu / redhat / systems / dockerd
Build packages
What format? RPMs / debs / tgzs
Push code
What tool? Ansible / Chef / CodeDeploy
Build Docker images
What registry? DockerHub / Quay / v1 / v2 / ECR
BUILD AND PACKAGING ADVENTURE: DOCKER IMAGES / PRIVATE REGISTRY
┌─────────────────────────────────────────┐ │ convox API │ │ POST /apps/httpd/build │ ┌──────────────────┐ │ │ │ │ │ ┌─────────────────────────────┐ │ ┌──────────────────┐ │ │ │ │ pull httpd-web:BLASTBUILD │◀───┼───────┤ │ ┌────────────────┐ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │◀───────────┤ ECS Instance │ │ ├─────┼─────▶│ pull rabbitmq │ │ │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ └────────────────┘ │ │ │ ┌─────────────────────────────┐ │ │ │ │ │ │ │tag httpd-rabbit:BQUWNCMIYZG │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ ┌────────────────┐ │ DockerHub │ │ ┌─────────────────────────────┐ │ │ │ │ │ │ or │ │ │ push $REGISTRY_HOST ├────┼──────▶│ │◀───────────┤ ECS Instance │ │Private Registries│ │ └─────────────────────────────┘ │ │ Convox Registry │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ └────────────────┘ │ ├─────┼─────▶│ pull debian │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ ┌────────────────┐ │ │ │ │ build Dockerfile │ │ │ │ │ │ │ │ │ └─────────────────────────────┘ │ │ │◀───────────┤ ECS Instance │ │ │ │ ┌─────────────────────────────┐ │ │ │ │ │ │ │ │ │ tag httpd-web:BQUWNCMIYZG │ │ │ │ └────────────────┘ │ │ │ └─────────────────────────────┘ │ │ │ │ │ │ ┌─────────────────────────────┐ │ │ │ └──────────────────┘ │ │ push $REGISTRY_HOST ├────┼──────▶└───────┬──────────┘ │ └─────────────────────────────┘ │ │ ▲ │ ┌─────────────────────────────┐ │ │ │ │ │tag httpd-worker:BQUWNCMIYZG │ │ ▼ │ │ └─────────────────────────────┘ │ ┌──────────┴───────┐ │ ┌─────────────────────────────┐ │ │ │ │ │ ... │ │ │ S3 │ │ └─────────────────────────────┘ │ │ │ │ │ └──────────────────┘ │ │ └─────────────────────────────────────────┘
BUILD AND PACKAGING R&D
Apr: AMIs are way too slow for CD workflow
May: DockerHub requirement doesn’t feel right
May: Running a Docker v1 registry with S3 store works!
Jun: v1 registry is flaky and impossible to debug and fix
Jun: v2 registry introduces lots of new stuff and incompatibilities
Oct: AWS announces ECR at re:Invent. Wait…
Dec: ECR GA
ECR MIGRATION ADVENTURE: CUSTOM RESOURCES
"RegistryRepository": { "Type": "Custom::ECRRepository", "Version": "1.0", "Properties": { "ServiceToken": { "Fn::GetAtt": [ "CustomTopic", "Arn" ] }, "Name": { "Ref": "AWS::StackName" } } },
"CustomTopic": { "Type": "AWS::Lambda::Function", "Properties": { "Code": { "S3Bucket": { "Fn::Join": [ "-", [ "convox", { "Ref": "AWS::Region" } ] ] }, "S3Key": { "Fn::Join": [ "", [ "release/", { "Ref": "Version" }, "/formation.zip" ] ] } }, "Handler": "lambda.external", "MemorySize": "128", "Role": { "Fn::GetAtt": [ "CustomTopicRole", "Arn" ] }, "Runtime": "nodejs", "Timeout": "30" } },
ECR MIGRATION ADVENTURE: PROGRAMMABLE INFRASTRUCTURE
$ convox update
Next deploy creates RegistryRepository Resource
Next deploy sees RegistryRepository Output pushes to it
Keep v1 registry / data around for rollbacks
ECR MIGRATION ADVENTURE: GLOBAL INFRASTRUCTURE
+ "RegionHasRegistry": { + "Fn::Equals": [ + { + "Ref": "AWS::Region" + }, + "us-east-1" + ] + },
ECR MIGRATION ADVENTURE: SUPPORT
watching for new regions
debugging ECR 500s on pushes
adding retry logic
adding error rate metrics
watching metrics around v1 usage
APP LOGS DECISION TREE
Use Amazon
What service? Kinesis / CloudWatch
Use 3rd Party
What vendor? Papertrail / Loggly / Splunk
Build your own pipeline
What system? syslog / logspout / logstash
App interface
stdout/stderr, log files, log drivers
APP LOGS ADVENTURE: AGENT, DOCKER APIS, KINESIS,
LAMBDA
┌──────────────────────────────────────────────────────────┐ ┌──────────────────┐ │ EC2 Instance in ECS Cluster │ │ app1 Kinesis │ │ │ │ ┌────────┐ │ ┌───────────────────────────────────────────┐ │ ┌──────────────┐ ┌──────────────────────────────────┐ │ ┌─┼───▶│shard 1 │ │──┐ │ Lambda w/ EventSourceMapping │ │ │ │ │ │ │ │ │ └────────┘ │ │ │ ┌──────────────────────────────────────┐ │ │ │ │ │ │ │ │ └──────────────────┘ │ │ │function(event, context) { │ │ │ │ app1 │ │ app2 │ │ │ │ │ │ event.records.forEach(function(r) { │ │ │ │ web.1 │ │ worker.1 │ │ │ │ │ │ winston.info(r.kinesis.data) │ │ │ │ │ │ │ │ │ └─┼▶│ }) │──┼────────▶┌───────────────┐ │ │ │ │ │ │ │ │ │ context.done() │ │ │ │ │ └──────────────┘ └──────────────────────────────────┘ │ │ ┌──────────────────┐ │ │} │ │ │ │ │ │ │ │ │ │ app2 Kinesis │ │ │ │ │ │ │ │ │ ┌─────────────────────┘ │ │ │ ┌────────┐ │ │ └──────────────────────────────────────┘ │ │ Syslog Server │ │ ▼ ▼ │ │ │ ┌─▶│shard 1 │ │ │ ┌────────────────────────────────┐ │ │ │ │ ┌────────────┐ ┌────────────┐─────────────┼───┘ │ │ └────────┘ │ │ │function(event, context) { ... }│──┼────────▶│ │ │ │ dockerd │◀─────────────│convox/agent│─────────────┼─────┼─┘ ┌────────┐ │ │ └────────────────────────────────┘ │ │ │ │ └────────────┘ └────────────┘─────────────┼─────┼───▶│shard 2 │ │ │ ┌────────────────────────────────┐ │ │ │ │ ▲ ┌────────────────────────────────────┐ │ │ └────────┘ │────┼─▶│function(event, context) { ... }│───────┼────────▶└───────────────┘ │ │ │GET docker /events (create) │ │ │ . │ │ └────────────────────────────────┘ │ │ ▼ │ GET ENV "Kinesis", "Process"│ │ │ . │ │ │ │ ┌────────────┐ │ GET Docker /logs?follow=1 │ │ │ . │ └───────────────────────────────────────────┘ │ │ ecs-agent │ │ PUT Kinesis /records │ │ │ ┌────────┐ │ │ └────────────┘ └────────────────────────────────────┘ │ │ │shard N │ │ │ │ │ └────────┘ │ └──────────────────────────────────────────────────────────┘ └──────────────────┘
APP LOGS R&D
Apr: Kinesis is rad
Oct: Kinesis + Lambda is really rad. Forward logs to Papertrail.
Nov: What about Kinesis to other systems?
Dec: What about CloudWatch Logs?
Jan: docker awslogs driver is great code
APP LOGS ADVENTURE: DOCKER/DOCKER INTEGRATION
APP LOGS ADVENTURE: INFRA AS JAVASCRIPT IN JAVASCRIPT
CONVOX PHILOSOPHY
Integrate, don’t invent
All-in on Docker Container and Image workflow
All-in on pure AWS solutions (where available)
Stub in missing pieces where not available
Build and exercise automated infrastructure migration paths
Remove custom pieces when no longer needed
[email protected] @nzoschke
_ _ _ _ | |_| |__ __ _ _ __ | | _____| | | __| '_ \ / _` | '_ \| |/ / __| | | |_| | | | (_| | | | | <\__ \_| \__|_| |_|\__,_|_| |_|_|\_\___(_)
I’m hiring...