Docker to the Rescue of an Ops Team

Docker to the rescue of an Ops Team

RachidZaroualiC.I.O SynoliaTwitter / Slack : Xinityrachid.zarouali@synolia.com

AgendaThis talk is about:

● Monitoring (a bit)● Private Registry● CI/CD● Security● Docker experience

Once upon a time An ops team starts a new project

Rebuild everything !June 2014A monitoring system from scratch“Microservices” orientedReplaceable parts

Some rules first !SimpleEfficientExtendable Python based

Components Collectd

Collectd proxy

Graphite

Grafana

Cabot (alerting)

Test your might !Python 2.6 along with python 2.7Different version of “some” librariesWhisper backend (I/O storm)Upstream repositories issues

Docker to the rescueSave our project

Docker ? way too soon !Barely no skillsUsed only to do some testingPretty serious concerns

Ok let’s gamble !Grow our docker fuWrite some DockerfilesBuild images locallySpawn PoC platform

…..

RUN echo "deb http://mirror.debian.ikoula.com/debian wheezy-backports main" >> /etc/apt/sources.listRUN apt-get -qq updateRUN apt-get -qqy dist-upgrade

RUN apt-get -qqy --force-yes install vim python-cairo gunicorn supervisor (...)RUN pip install whitenoise txamqp whisper==0.9.13 carbonateRUN pip install --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/lib" carbon==0.9.13RUN pip install --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/webapp" graphite-web==0.9.13

ADD conf/nginx.conf /etc/nginx/nginx.confADD conf/supervisord.conf /etc/supervisor/conf.d/grafana.conf

ADD initial_data.json /var/lib/graphite/webapp/graphite/initial_data.jsonADD conf/local_settings.py /var/lib/graphite/webapp/graphite/local_settings.pyADD conf/carbon.conf /var/lib/graphite/conf/carbon.confADD conf/storage-schemas.conf /var/lib/graphite/conf/storage-schemas.confRUN mkdir -p /var/lib/graphite/storage/whisperRUN touch /var/lib/graphite/storage/graphite.db /var/lib/graphite/storage/indexRUN chmod 0775 /var/lib/graphite/storage /var/lib/graphite/storage/whisperRUN python /var/lib/graphite/webapp/graphite/manage.py syncdb --noinput --pythonpath=/var/lib/graphite/webapp/graphite --settings=settingsRUN chmod 0664 /var/lib/graphite/storage/graphite.dbRUN chown -R www-data /var/lib/graphite/storage

…..

WHAT ???

Container = OS …. Wait !

Too many layers ( 121+ layer issue)

Build time …. (20 to 30 minutes at best)

Huge Images (800+ Mo)

Unnecessary tools and libs

Bye Bye !!!!

We can do better !Apply best Practices (@abbyfuller)

Implement simple CI/CD

Dockerfile Linting

Build a private registry

Deal with security concerns

…

RUN echo "APT::Install-Recommends false;" >> /etc/apt/apt.conf.d/00recommends \ && echo "APT::Install-Suggests false;" >> /etc/apt/apt.conf.d/00recommends \ && echo "APT::AutoRemove::RecommendsImportant false;" >> /etc/apt/apt.conf.d/00recommends \ && echo "APT::AutoRemove::SuggestsImportant false;" >> /etc/apt/apt.conf.d/00recommends

ENV DEBIAN_FRONTEND noninteractiveENV GRAPHITE_VERS 0.9.13

RUN apt-get -qqy update \ && apt-get -qqy install python-cairo gunicorn git python2.7-dev wget ca-certificates python-flup expect sqlite3 libcairo2 libcairo2-dev pkg-config nodejs sqlite3 memcached python-ldap make gcc libffi-dev

RUN wget https://bootstrap.pypa.io/get-pip.py \ && python get-pip.py \ && pip install --no-cache-dir --upgrade setuptools \ && pip install --no-cache-dir django django-admin-tools \ && pip install --no-cache-dir whitenoise txamqp whisper==${GRAPHITE_VERS} carbonate \

&& pip install --no-cache-dir --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/lib" carbon==${GRAPHITE_VERS} \

&& pip install --no-cache-dir --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/webapp" graphite-web==${GRAPHITE_VERS}

RUN apt-get purge gcc make python2.7-dev libcairo2-dev libffi-dev python2.7-dev pkg-config -qqy \ && apt-get clean\ && apt-get autoremove -qqy \ && rm -rf /root/.cache /var/lib/apt/lists/* /tmp/* /var/tmp/*

...

Best practices :)Few image layers ( < 20 )Small image (~ 400Mo)Lower footprint (100Mo)Faster build time (~5Min)

We did it , we did it Yeah !

CI/CD Diagram

CI/CD Recipe

build: image: registry.synolia.com/synolia/dockerunitest:latest

publish: docker: repo: synomon_datastor tag: $${BRANCH/master/latest} file: Dockerfile insecure: true when: repo: synolia/systeam-monitoring_datastor branch: [develop, master]

notify: hipchat: from: "synoci" room_id_or_name: "$$ROOM_ID" auth_token: "$$AUTH_TOKEN" notify: true when: success: false failure: true

Dockerfile Linting

Build

Push

Notify (fail only)

Docker to the rescue IIThe return of the hero moby

This isn’t over yet !Docker udp issuesTricky iptable filteringUnstable data volumeConfiguration management

Round 2: FIGHT !Metric proxy (Collectd) on the host!!Simplify iptables rulesMount directories (metrics)

A new path opensTo a brighter future

Epic loots !No more dependency issuesReplaceable and movable partsGreater security levelClustering ready (Swarm/K8S)

Lessons learnedCaution when using udp IPv4Config files out of the containerDon’t use env variables (security)Use (abuse) automation

What’s next ?Greater Docker challenges

Many rooms to growReduce (even more) Image sizeSign Images (notary to the rescue)Vulnerability scanningImplement rolling upgrades

New docker based projectsMigrate Development platform (2015) Swarm clustering (*)Full scale Docker (*)(*) Work In Progress

Thank You DockerCon !PS: don’t forget to rate my talk :)