Embed Size (px)
Citation preview
BLOGLOVIN’
Using SaltStack to orchestrate microservices in application
containers
Love Nyberg Site reliability & Automation Engineer
#SaltConf15
BLOGLOVIN’
#SaltConf15
tens of millions of monthly unique visitors hundred of millions requests per day couple of million stories parsed per day
Hybrid DC/Cloud - Bare metal servers - Cloud VM’s
All those buzz words!
• Configuration management • Continues Integartion • Containers • Orchestration • Micro services
#SaltConf15
Why containers?
Build
#SaltConf15
Ship Run
#SaltConf15
#SaltConf15
All those tools!
Legacy!
#SaltConf15
#SaltConf15
Evolution of container management at bloglovin?
Phase 1 Rolling deploys with static amount
Phase 2 Green/Blue deploy with dynamic pillar
#SaltConf15
Build
#SaltConf15
Ship Run
Architecture
#SaltConf15
LB/Cache
Ship - applications
DB/Queues
Salt master
Jenkins
Registry
Phase 1: Rolling deploys with static amount
Static amount of containers on each ship
Rolling deployes on each ship
Drawbacks no healtchecks serving diffrent version of code during deploy
#SaltConf15
Building images
#SaltConf15
Dockerfile Jenkins Registry
Building images
#SaltConf15
base image
application image
application code
Ship images
#SaltConf15
Run containers
#SaltConf15
Run containers
#SaltConf15
{% set name = 'node-demo' %} {% set registryname = 'jacksoncage' %} {% set tag = salt['pillar.get']('imagetag', "latest") %} {% set containerid = salt['grains.get']('id') %} {% set hostport = '808' %} {% set hostip = grains['ip_interfaces']['eth0'][0] %} {% set tag = salt[‘pillar.get']('containers:imagetag', "latest") %} {% set noofcontainers = range(10) ‰}
State file: 1. Setting variables
Run containers
#SaltConf15
{{ name }}-image: docker.pulled: - name: {{ registryname }}/{{ name }} - tag: {{ tag }} - force: True
State file: 2. Pulling docker image
Run containers
#SaltConf15
{{ name }}-stop-if-old-{{ no }}: cmd.run: - name: docker stop {{ containerid }}-{{ name }}-{{ no }} - unless: docker inspect --format '{{ .Image }}' {{ containerid }}-{{ name }}-{{ nr }} | \ grep $(docker images --no-trunc | grep "{{ registryname }}/{{ name }}" | awk '{ print $3 }') - require: - docker: {{ name }}-image
State file: 3. Stop container if newer image exists
Run containers
#SaltConf15
{{ name }}-remove-if-old-{{ no }}: cmd.run: - name: docker rm {{ containerid }}-{{ name }}-{{ no }} - unless: docker inspect --format '{{ .Image }}' {{ containerid }}-{{ name }}-{{ nr }} | \ grep $(docker images --no-trunc | grep "{{ registryname }}/{{ name }}" | awk '{ print $3 }') - require: - cmd: {{ name }}-stop-if-old-{{ no }}
State file: 4. Removing container if newer image exists
Run containers
#SaltConf15
{{ name }}-container-{{ no }}: docker.installed: - name: {{ containerid }}-{{ name }}-{{ no }} - hostname: {{ containerid }}-{{ name }}-{{ no }} - image: {{ registryname }}/{{ name }}:{{ tag }} - ports: - "8080/tcp" - environment: - EXECUTER: "forever -f start" - APP: "index.js" - require_in: {{ name }}-{{ no }} - require: - docker: {{ name }}-image
State file: 5. Installing container
Run containers
#SaltConf15
{{ name }}-{{ no }}: docker.running: - container: {{ containerid }}-{{ name }}-{{ no }} - port_bindings: "8080/tcp": HostIp: "{{ hostip }}" HostPort: "{{ hostport }}{{ no }}"
State file: 6. Start container
Run containers
#SaltConf15
salt minion1 state.sls containers.applications.node-demo 'pillar={ containers: { imagetag: 1.0.0 } }'
Run: command line
Run containers
#SaltConf15
curl -H 'X-Salt-Deploy-Key: QmpaDa8T3UdFVBUHWo@T' \ -H 'Content-Type: application/json' \ -d '{"application":"api","container":"api-phase1","node":"ship01","tag":"1.0","amount":"2"}' \ http://127.0.0.1:8000/hook/api/deploy/phase2/success
salt-call event.fire_master '{"salt-node": "ship01", "application": "api", “container": \ "api-phase1", "tag": "1.0"}' deploy/containers
Run: via api/reactor
Phase 2: Green/Blue deploy with dynamic pillar
• Deploy a new containers before killing the old once
• Use healtchecks
• Use dynamic updated pillar data with etcd
#SaltConf15
Run containers
#SaltConf15
Run containers
#SaltConf15
Reactor sls
1. Sync states 2. Update etcd with tags, amount etc 3. Pull docker image 4. Deploy unactive containers to cluster 5. Test unactive containers and add to etcd 6. Update haproxy using etcd 7. Close down unactive containers
Demo
#SaltConf15
Phase 1: https://github.com/jacksoncage/docker-node-demo
Phase 2: https://github.com/jacksoncage/saltconf-container-demo
Container madness!
Images
Logging
Monitoring
#SaltConf15
