DevOpsDays Paris 18-19 avril 2013 - blog.groupe-sii.com€¦ · DevOpsDays Paris 18-19 avril 2013 ... Quality for us = Quality of product + quality of support DevOps engineers supported

Page 1 www.groupe-sii.com 02/07/2013

DevOpsDays Paris 18-19 avril 2013

G. Bloquel – S. Révéreault

Page 2

DevOpsDays :

Série d’événements dans des grandes villes internationales

Modèle unique : conférences, « ignite talks », Open Spaces (+

demos)

Co-organisation avec Patrick Debois

Paris : 1er événement français

Sponsors :

Introduction

Page 3

Monitoring Infrastructure As A Service

What we do : write software

What we sell : service

No marketing, just word of mouth

Quality for us = Quality of product + quality of support

DevOps engineers supported by DevOps engineers

Culture

engineering culture == build && understand

answer every single question, give as much insight as possible

"is the problem solved?", "Does the answer make sense?"

Every engineer does customer support 1 week at a time. => we learn

Split roles in support : interrupt driven / problem solve

Conf #1 : CustomerOps: a culture of visibility and metrics applied to customer support

Alexis Le-Quoc - @alq - Co-founder and CTO of Datadog

Page 4

Automation :

Identify channels for

answering questions : FAQ, IRC, in-product chat, email, ...

– Desk, #freenode, olark

reaching out individuals : auto-email (emails based on what you're doing)

– Intercom.io, customer.io

broadcasting : status page, twitter

– Stashboard + gae, twitter

sharing: code, API Docs

– Github, jekyll + S3

observing : analytics

– kissmetrics

Reactive + Proactive approach


Page 5

Measurement :

Monthly metrics

New cases: Quality

Reopened case ratio: Quality

Time 1st response : Engagement

Time to resolution : Engagement

1st contact resolution %: Engagement

Volume per channel : Productivity

Impact of support on sales : Sales

Sharing :

Contribution aux DevOpsDays.

Questions ouvertes :

What can we improve ?

How can I talk to my customers ?


Page 6

Q/R :

How do you prevent your support team from optimizing metrics ?

We don't really have the problem.

Would this approach work for "normal" end users (not technical ones) ?

I think that still applies, filters might be different.

Links :

Description : http://www.devopsdays.org/events/2013-

paris/proposals/CustomerOps/

Slides : https://speakerdeck.com/alq/customerops

Vidéo : http://vimeo.com/66153903

Twitter : @alq


http://www.devopsdays.org/events/2013-paris/proposals/CustomerOps/




https://speakerdeck.com/alq/customerops

https://speakerdeck.com/alq/customerops

http://vimeo.com/66153903


Page 7

When I started : developping on a LAMP stack, with a strict Dev/Ops

separation.

=> It wasn't even that bad… until projects got special

One of the most efficient teams was a dev, an admin and a cup of coffee.

« git push developer mindset/devops »

Explications du fonctionnement autour d'un outil de récupération et

d'encodage de vidéos (architecture, problèmes, changements).

Infrastructure as code, mais aussi code for infrastructure.

Importance des métriques et des logs dans les systèmes complexes

la plupart des équipes de dev l'implémentent trop tard, alors que ça doit être

une des premières choses faites.

Conf #2 : How ops improved my dev

Florian Gilcher - @agorak - Ruby Dev, consultant

Page 8

Importance pas des moindres, l’utilisation d’outils interne peut nous faire

économiser beaucoup de temps

Platform refactorings

Code reviews

Pas assez de gens sont qualifiés pour en faire

Moins de permission, plus de choses serait faite

Elargisser la connaissance dans vos équipes

Conclusion

Les equipes avec une culture « devops »

Peuvent s’occuper de problème plus complexe

Peuvent trouver des solutions alternatives à des problèmes


Page 9

Quelques citations :

« Devops-minded teams can cope with missing team members easier »

« My ugliest piece of code ran 1,5 years in production without a change »

« The Devops mindset takes away a lot of friction »

Links :


paris/proposals/DevelopmentGainsThroughOps/

Slides : https://speakerdeck.com/skade/how-devops-improved-my-dev


Twitter : @agorak


http://www.devopsdays.org/events/2013-paris/proposals/DevelopmentGainsThroughOps/



https://speakerdeck.com/skade/how-devops-improved-my-dev












Page 10

What is a project

A project is a set of activities and actions with a specific need in specified time and a budget

What is a product

A product is a result a creative human activity

Spot the differences between product and project, and the IT involvement.

Project is composed action, activity, specific need, time, budget

Product is composed creative activity, satify need, client

Project fail, product succeed, how to make projects succeed like products?

Don't forget product team are users too, and they have good ideas.

Conf #3 : Product or Project ?

Rémy-Christophe Schermesser, @el_picador, Octo Technology

Page 11

Team project

Business analyst (MOA)

Project manager

Team product

Developper

Marketing

Designer

Product Manager

Give responsability


Page 12

What makes a product succeed ?

Build :

Make the Minimum Viable Product (MVP) , and go to production ASAP

Help end users to change (for instance, quick tutorials). Voir chardin.js => librairie javascript pour

faire du micro-tutoriel. (Tools base sur Jquery permettant d’ajouter une légende par superposition)

Measure

Users lie to survey

Build things and then ask people for their feedback (Ford : Si j'avais demandé aux gens ce

qu'ils voulaient, ils m'auraient dit "des chevaux plus rapides").

Get out of the building


Learn

Build

Measure

Page 13

Learn :

Fail to learn

Think product and do projects.

Links :

Description : http://devopsdays.org/events/2013-

paris/proposals/ProduitOuProjet/

Slides : https://speakerdeck.com/elpicador/how-products-can-improve-projects


Twitter : @el_picador


http://devopsdays.org/events/2013-paris/proposals/ProduitOuProjet/




https://speakerdeck.com/elpicador/how-products-can-improve-projects












Page 14

Why do you need DevOps ?

Business needs velocity.

V-cycle : time consuming, tunnel effect. Slower and Riskier than you can estimate

(voir étude 2 profs Oxford sur la réussite des projets de développement)

1 projet / 6 dépasse le cout attendu par 3

Our brain sees the world through Bell Curves (livre : the black swan), but

complex systems do not act like bell curves

Small iterations are important (Divide and Conquer).

Lean startup Devops

=> DevOps is the result of lean thinking

Conf #4 : Transforming devs into devops

Fabrice Bernhard, Fondateur et Directeur Technique Theodo

Page 15

Etapes :

1) Dev to DevOps friendly

2) DevOps friendly to DevOps

Dev to Devops friendly

Typical junior dev knows nothing

about Ops

Learnt java at university

Uses windows because of the

games

Has already installed linux because

he's curious


1st step: improove developpers

Developping under Linux

Git versionning

Correct git branching

Scrum method

Unit and functionnal tests

Bring Devs to DevOps :

Shell scripting ? forget it

Capistrano ? too magic

Fabric ? good compromise

Page 16

=> Every project has a deploy.py script in a "devops" folder.

Give the chance for Devs to play with a server

Acquire experience, need sandbox dev, simple to reset

IaaS for dev sandbox

OK for dev, less for testing : clients complained that the testing platform was unstable

Scripted provisioning must stay simple : Dev need to install packages and modify

config files on their server

Chef-server / puppet-server : designed for clusters

Chef-solo : a "who wirites the most magical ruby" contest

Puppet/Chef modules : too much abstraction, not always work "out of the box"

Fabtools : easy and developer-friendly

Puppet apply without modules (currently testing)

Use Vagrant to create the dev environment and use chef / puppet


Page 17

Devops friendly to DevOps

Devs + ops tools = devops ? no

Deep cultural differences between dev and ops

Pair devopsing as an efficient teaching solution

sysadmin should always pair with a dev when working on a project

scrum compatible

hire devs that are eager to learn stuff

Issues and constraints for developing fast

TDD

Performance

Provisioning

Scaling

Backups

How do you make devs focus on performance ?

Make it part of the DONE definition

Include performance solutions in the standard provisioning (for instance use Varnish in dev if you

use it in prod)

AND do visible performance graphs


Page 18

Make scaling cool and intellectually challenging

The power of NoSQL : devs have to think the model in a scalable way

IaaS is and infrastructure with an API... cool !

How do you bring devs to feel responsible for production (http://xkcd.com/705/ )

Why do ops care, and not dev : not our job

What about backup and server monitoring ?

SaaS (Newrelic, Pingdom, Idera ServerBackup)

for bigger needs, you need a sysadmin

Links :


paris/proposals/TransformerDesDevsEnDevops/

Slides : 404 Not Found


Twitter : @theodo


http://www.devopsdays.org/events/2013-paris/proposals/TransformerDesDevsEnDevops/





Page 19

DevOps Productivity Survey 2013 (RebelLabs)


http://zeroturnaround.com/labs/rebel-labs-release-it-ops-devops-productivity-report-

2013/

DevOps est mieux que l’IT traditionnelle (métriques à l’appui)

Performance perpétuelle – Octo


http://fr.slideshare.net/henrit/perf-devops-day

Pourquoi mesurer la performance en continu ?

Deployer en continu des VMs

Test de charge (Gatling)

Casser le build si performance KO

Ignite #1 :



http://zeroturnaround.com/labs/rebel-labs-release-it-ops-devops-productivity-report-2013/


























Page 20

Lost paradise of DevOps (Serena)


DevOps is not new !

Comparaisons « bibliques » : la tour de Babel, le miracle de la mer rouge, …

Project Raindrops


http://projectraindrops.net/login

Service de création automatique d'images de VMs pour différentes infra (Aws, vagrant)

What if DevOps was invented by Coca Cola ? P. Debois


http://fr.slideshare.net/devopsdays/what-if-devops-was-invented-by-coca-cola

« Don’t take devops too serious, use with caution »

Ignite #1 :

























Page 21

Continuous Integration for infrastructure

Quels outils ?

Jenkins, mais est-il bien adapté

Puppet, Chef, CFEngine, …

Packaging : RPM, Deb, MSI, Jar/War, Tar.gz, …

Que doit-on emprunter aux méthodos de développement ?

Tests unitaires sur les recettes de déploiement

Smoke test

Factorisation du code (exemple : Specs RPM)

Conclusion :

L’outillage n’est pas mûr

Méthodos encore obscures pour la partie infra

Open Spaces

Page 22

Introducing DevOps in a traditionnal organisation / Tips and tricks to spread

culture

Par où commencer :

Trouver les leviers (douleurs et comment DevOps peut les résoudre)

Trouver les personnes de bonne volonté

Fixer les objectifs

Projets pilotes

Quelques trucs :

Faire travailler Dev et Ops sur des douleurs communes

– Logs

– On duty call

Attention à ne pas établir une relation client / fournisseur entre Dev et Ops

Faire contribuer les Ops à la création de la liste de métriques

Open Spaces

Page 23

jRebel / LiveRebel

jRebel : voir les changements en temps réel sur du code développé

en Java

LiveRebel : gestion de configuration et de déploiement

Un agent prend la main sur le port en écoute de Tomcat

En cas de déploiement progressif, l’agent va se charger des

redirections éventuelles de trafic

CFEngine

Démo de suppression d’un élément et reconfiguration

automatique de la plateforme.

Démos

Page 24

Présentation de GDS (sous-organisation de "Cabinet Office", service public Anglais).

But : "Digital services so good that people prefer to use them".

Focus on user needs

Ship Fast

Measure everything

Manifeste: "Revolution, not Evolution"

Build a centre of excellence

Fix publishing

Fix transactions

Go wholesale

Gov.uk : site d'information sur l'activité du gouvernement britanique

"Nobody ever got fired for choosing open source“

Le site est publié sur Github : github.com/alphagov

Conf #5 : How we release software for GOV.UK

Kushal Pisavadia - GDS

Page 25

Outils utilisés pour le déploiement :

Capistrano au départ

Ensuite Jenkins

Commit to master, kick off a build, test, deploy

Build screen

Un bel exemple de User Story pour décrire une exigence de performance

Configuration management testés :

Opscode

PalletOps

Puppet => choisi (un blog post décrit la comparaison)

We gave all the developers sudo access (we were working on a beta).


Page 26

Toutes les pages d'info peuvent être récupérées en json.

1-click deploy : utilisation d'un "gros bouton" pour symboliser la MEP.

Simplify with abstractions (exemple : vous connaissez le script, mais vous ne

connaissez pas la physique quantique. Pourtant, vos scripts sont exécutés par

des ordinateurs qui utilisent des transistors qui s'appuient sur de la

mécanique quantique).

Getting ready for the October release => being the home page of a nation.

Enforced rotation of developers

Mise en place d'une application pour gérer les modifications


Page 27

Retour sur la faille Ruby on Rails de janvier => application patchée en 1h30

(nobody was "on call"... they just cared).

https://www.gov.uk/service-manual/operations/devops.html

4 hours between code change and production (including manual validation

by product team)

15 to 20 deploys a day

Links :


paris/proposals/HowWeReleaseSoftwareForGOV.UK/


Vidéo : Bientôt sur http://vimeo.com/user9086015

Twitter : @KushalP






http://www.devopsdays.org/events/2013-paris/proposals/HowWeReleaseSoftwareForGOV.UK/




http://vimeo.com/user9086015


Page 28

Visibility.

Design, Building, but mostly corrections

=> How to improve ?

Break Silo Approach: 30% dev and 70% ops

Agile Approach 70% build et 30% correctino

We want lower defect rates.

We want to make informed decisions.

=> Visibility : extracting meaningful state data from heterogenous event sources, over

time

- Meaningful (toward business)

- State data

- Heterogeneous (everyone is involved in creating data)

- Over time

Conf #6 : Le territoire et la carte - une histoire de visibilité

Pierre-Yves Ritschard, @Pyr, developeur OpenBSD

Page 29

How does it help my system's life cycle.

"The map is not the territory."

Break out of our mental model.

Less maps, more territory... or at least better maps.

Systems are increasingly complex.

En 2000 : LAMP (2 serveurs). Visibility : MRTG graphs

En 2012 : HA Proxy, NGinx, Cassandra, Redis, MySQL, ... (27 serveurs). Visibility :

we're still looking at the same metrics... WRONG !

Figure out key metrics :

We need appropriate tooling

Metrics & Events : correlation accross : system, components, software


Page 30

The event stream approach :

Plenty of small producers, and few big consumers

Anything that happen or moves : normalize & stream

Aggregate : compute compound metrics

Correlate : find trends

Decide : track, alert, ignore, scale

Implementing : on premise, saas, or in between ?

SaaS : Loggly, papertrail, librato, datadog, ...

On Premise : collectd, logstash, graphite, statsd, riemann


Page 31

The path to visibility :

Find key metrics

Find the right tools

Rely on an event stream

Involve everyone

Challenge your mental model

Hopefully improves quality and lower perfect rates in the process

Links :


paris/proposals/TerritoireCarte/

Slides : https://speakerdeck.com/pyr/map-and-territory-a-story-of-visibility


Twitter : @Pyr


http://www.devopsdays.org/events/2013-paris/proposals/TerritoireCarte/



https://speakerdeck.com/pyr/map-and-territory-a-story-of-visibility















Page 32

Trap #1 : Think it's only tooling

Tools, Processes, culture

Trap #2 : Start with the wrong tooling strategy

Build your own solution ?

Pick up a deployment tool ?

Appliance images ?

Adopt PaaS approach ?

Conf #7 : The 10 major traps to avoid in order to succeed your DevOps transition Alain Delafosse, Consultant Objet Direct et Directeur Technique KelKoo Eric Mattern ,Manager Open Source Objet Direct

Cloner Configurer Developper Packager Configurer Deployer

Applications Infra / OS

Page 33

Trap #3 : Think only server deployment

Configuration management (OS & software configuration items)

Software deployment (+ Software components & middleware)

Platform deployment (+ Infra config & provisioning, Deployment strategy)

Trap #4 : Trying to fully automate data & database upgrades

Tricky to automate

Try to uncouple database changes from software upgrades

Defer database structure change

New database schema

NOSQL database schemaless

Conf #7 : The 10 major traps to avoid in order to succeed your DevOps transition

Page 34

Trap #5 : Rebranding teams/persons as DevOps

Don't

rename teams

setting DevOps as title or function

create a new team

Do

Forget the name

empower your teams to foster the transition

Trap #6 : Underevaluate the cultural shift

Collaboration

Long and difficult journey

DevOps cultural aspects : sense of continuous improvement, collaboration level,

transparency

Re-use Agile transitions practices


Page 35

Trap #7 : If you don't measure it, you can't value it

Define goals -> Find metrics -> Expose KPIs

Contextualize metrics to your goal and context

Evaluate progression & share the success

Deployement rate and delay

Release quality, number of rollback

Effort and delay for conf update

Trap #8 : Forget continuous improvement loops

DevOps is a endless journey

Setup a continuous improvement loop : tools and

processes


Capture feedbacks

Formalize and categories

Backlog Milestone and

priority

Improve implementation

and deploy

Page 36

Trap #9 : Focus only on deployment process

There are other processes / areas where collaboration between dev and ops make sense.

Monitoring

– Common dashboard using dev, ops and BIZ

– Log management

Performance management

– Integration continous

Incident / problem management

BCP (Business Continuity) / DRP (Disaster Recovery) design

Security management

Trap #10 : Conflict with ITIL

Some production environments do require strong service management

DevOps doesn't enforce a specific organization of processes

DevOps will seed the ITIL processes by bringing new tools and new practices

Change management


Page 37

Links :


paris/proposals/Les%2010%20pie%CC%80ges%20principaux%20a%20e%CC%81v

iter%20pour%20re%CC%81ussir%20votre%20transition%20DevOps/



Twitter : @AlainDelafosse


http://www.devopsdays.org/events/2013-paris/proposals/Les 10 pie%CC%80ges principaux a e%CC%81viter pour re%CC%81ussir votre transition DevOps/





Page 38

Testing.

When working with puppet, a lot of manual testing => unit tests

on Puppet

A voir : Fizzgig (librairie de test de recettes Puppet).

Managing Complexity (Diego Zamboni - CFEngine)

Abstraction, Perspective. Our tools are not ready, but we are

working on it.

Aeolus (R. Di Cosmo - Inria)

Deployment plan on a Cloud. Zephyrus : an automatic tool for

creating a full system of configurations.

Ignite #2

Page 39

DevOps coaching :

Tips :

Scrum, Scrum de scrum

Le Scrum Master doit être responsable de l’implication des ops

Trouver les leviers, trouver les points de motivation

Charte d’utilisation des métriques

Attention à la limite coaching / audit

Equipes pilote, évangélisation par les pilotes

Constat :

On manque de méthodologie

S’appuyer sur agile, scrum

Open Space

Page 40

Ruddler

GUI Tool using CFEngine

No need to write own files

use generic template through GUI

Rollback possible thanks to historic configuration

DEMO #2

Page 41

Beaucoup d’échanges autour de la culture

Une vraie effervescence

Intérêt des grands acteurs

L’outillage est incomplet

Adapter les concepts

Conclusion

Documents

DevOpsDays Paris 18-19 avril 2013 - blog.groupe-sii.com€¦ · DevOpsDays Paris 18-19 avril 2013 ... Quality for us = Quality of product + quality of support DevOps engineers supported