No, we can not do Continuous Deployment

Kris Buytaert

@krisbuytaert

Kris BuytaertKris Buytaert•I used to be a Dev,I used to be a Dev,

•Then Became an OpThen Became an Op

•Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source Consultant @Consultant @inuits.euinuits.eu

•Everything is an effing DNS ProblemEverything is an effing DNS Problem

•Building Clouds since before the bookstoreBuilding Clouds since before the bookstore

•Some books, some papers, some blogsSome books, some papers, some blogs

•Evangelizing devopsEvangelizing devops

•Organiser of #devopsdays, #cfgmgmtcamp, Organiser of #devopsdays, #cfgmgmtcamp, #loadays, ….#loadays, ….

What's this devops What's this devops thing anyhow ? thing anyhow ?

World , 200X-2009World , 200X-2009Patrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, Jezz Patrick Debois, Gildas Le Nadan, Andrew Clay Shafer, Kris Buytaert, Jezz Humble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, and Humble, Lindsay Holmwood, John Willis, Chris Read, Julian Simpson, and

lots of others ..lots of others ..

Gent , October 2009Gent , October 2009

Mountain View , June 2010Mountain View , June 2010

5th aniversary 2 years ago in Gent5th aniversary 2 years ago in Gent

........

C(L)AMSC(L)AMS

•CultureCulture•(Lean)(Lean)•AutomationAutomation•MeasurementMeasurement•SharingSharing

Damon Edwards and John WillisDamon Edwards and John Willis

Gene KimGene Kim

NirvanaNirvanaAn “ecosystem” that supports continuous delivery, from An “ecosystem” that supports continuous delivery, from infrastructure, data and configuration management to infrastructure, data and configuration management to business.business.

Through automation of the build, deployment, and testing Through automation of the build, deployment, and testing process, and improved collaboration between developers, process, and improved collaboration between developers, testers, and operations, delivery teams can get changes testers, and operations, delivery teams can get changes released in a matter of hours — sometimes even minutes–no released in a matter of hours — sometimes even minutes–no matter what the size of a project or the complexity of its code matter what the size of a project or the complexity of its code base.base.

Continuous Delivery , Jez HumbleContinuous Delivery , Jez Humble

Continuous IntegrationContinuous IntegrationContinuous integration (CI) is the practice, in software engineering, of merging Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day. It was all developer working copies with a shared mainline several times a day. It was first named and proposed as part of extreme programming (XP). Its main aim is first named and proposed as part of extreme programming (XP). Its main aim is to prevent integration problems, referred to as "integration hell" to prevent integration problems, referred to as "integration hell"

(WikiPedia)(WikiPedia)

Does the app you are deploying still work ?Does the app you are deploying still work ?

Did you break your infrastructure ?Did you break your infrastructure ?

CD vs CDCD vs CD

•Continuous Continuous

DeliveryDelivery•Knowing you can Knowing you can •One buttonOne button•Human decisionHuman decision

•ContinuousContinuous

DeploymentDeployment•Constantly doing itConstantly doing it•No More buttons No More buttons •Machine decisionMachine decision

devops (<)> continuous delivery devops (<)> continuous delivery

Culture,Culture,automation, automation,

Measturement,Measturement,

sharingsharing

How many times a day ?How many times a day ?

•10 @ Flickr10 @ Flickr•Deployments used to be painDeployments used to be pain•Nobody dared to deploy a siteNobody dared to deploy a site•Practice makes perfectPractice makes perfect•Knowing you can vs constantly doing it Knowing you can vs constantly doing it

" Our job as engineers (and ops, dev-ops, " Our job as engineers (and ops, dev-ops, QA, support, everyone in the company QA, support, everyone in the company actually) is to enable the business goals. actually) is to enable the business goals. We strongly feel that in order to do that We strongly feel that in order to do that you must have the you must have the ability to deploy code ability to deploy code quickly and safely.quickly and safely. Even if the business Even if the business goals are to deploy strongly QA’d code goals are to deploy strongly QA’d code once a month at 3am (it’s not for us, we once a month at 3am (it’s not for us, we push all the time), having a push all the time), having a reliable and reliable and easy deploymenteasy deployment should be non- should be non-negotiable." negotiable." Etsy Blog upon releasing DeployinatorEtsy Blog upon releasing Deployinator

http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/

Whats in it for you ?Whats in it for you ?•Faster time to marketFaster time to market•Features go live in hours vs yearsFeatures go live in hours vs years•In a more safe (Secure)In a more safe (Secure)•Reliable fashionReliable fashion•Fully automatedFully automated•More happy More happy {customers,developers,managers,investors}{customers,developers,managers,investors}

Pushing to production 5 times a day .. Pushing to production 5 times a day ..

Are you out of your mind ? Are you out of your mind ?

vs vs

Sure we do that Sure we do that

This is not Continuous DeploymentThis is not Continuous Deployment

•@stahnma @stahnma @#devopsdays Ohio@#devopsdays Ohio

It is Pink SombreroIt is Pink Sombrero

But people sayBut people say

•It's too riskyIt's too risky•We can't , our We can't , our users don't want users don't want to ...to ...•But <insert But <insert favourite procedural favourite procedural framework >framework >•But auditors/ But auditors/ ComplianceCompliance•It's too expensiveIt's too expensive

It's too riskyIt's too risky•We deployed 6 months ago, it was We deployed 6 months ago, it was painfull, we needed 3 weeks aftercarepainfull, we needed 3 weeks aftercare

•There's 3576 changes in the new There's 3576 changes in the new deploy, we have no clue what caused deploy, we have no clue what caused this problemthis problem

•We need 20 people in a room for 8-12 We need 20 people in a room for 8-12 hourshours

•I have no clue why I wrote that line of I have no clue why I wrote that line of code 3 months agocode 3 months ago

•The person who wrote this left 2 The person who wrote this left 2 weeks agoweeks ago

•Ooops we forgot to delete that feature Ooops we forgot to delete that feature they don't want anymore.they don't want anymore.

•We deploy automaticaly, We deploy automaticaly,

•I clearly remember what we fixed I clearly remember what we fixed yesterday yesterday

•And that's the only thing that has And that's the only thing that has changed in the last commitchanged in the last commit

•The person who wrote the code is still The person who wrote the code is still in the buildingin the building

•We really need this feature now, we We really need this feature now, we can remove it later can remove it later

Every commitEvery commit with with successful testssuccessful tests will will automaticallyautomatically be be deployed deployed production production

Every commitEvery commit with successful test with successful test will automatically be deployed to will automatically be deployed to production production

Version control Version control

Who, changed what, why and whenWho, changed what, why and when

Every commit with Every commit with successful testssuccessful tests will automatically be deployed to will automatically be deployed to productionproduction

Automated testing strategy, is keyAutomated testing strategy, is key

Successful tests, no bypassing of the Successful tests, no bypassing of the teststests

Test all the thingsTest all the things•Unit testsUnit tests•Integration TestsIntegration Tests•System TestsSystem Tests•Acceptance TestsAcceptance Tests•Security TestsSecurity Tests•Performance TestsPerformance Tests

•Regression TestsRegression Tests•Functional TestsFunctional Tests

24

-

While Culture = OKWhile Culture = OK

- Write code- Write code

- Keep pipeline running- Keep pipeline running

- Stop the pipeline on failure- Stop the pipeline on failure

- Fix the pipeline- Fix the pipeline

- Don't go home on a broken pipeline- Don't go home on a broken pipeline

PromotionsPromotionsWhen you don't trust your test yet, you'll When you don't trust your test yet, you'll

put in manual checkpoints where humans put in manual checkpoints where humans verify and approve, while adding more verify and approve, while adding more

tests.tests.

Beware of the unpromoted builds : Beware of the unpromoted builds :

They risk quickly ending up in big, They risk quickly ending up in big, delayed, problematic releases again.delayed, problematic releases again.

Testing = MonitoringTesting = Monitoring•Add it to the monitoring frameworkAdd it to the monitoring framework•Add collection toolsAdd collection tools•Add check definitionsAdd check definitions•Update the monitoring tool configUpdate the monitoring tool config•Deploy a host,Deploy a host,

FULLY AUTOMATEDFULLY AUTOMATED

Every commit with successful tests Every commit with successful tests will will automaticallyautomatically be deployed to be deployed to productionproduction

Automate all the things !Automate all the things !

No humans involved, No humans involved,

Less error proneLess error prone

Less boringLess boring

Every commit with successful tests Every commit with successful tests will automatically be will automatically be deployed to deployed to productionproduction

Deployed code does not mean Deployed code does not mean enabled feature.enabled feature.

We can't , our ...We can't , our ...•Marketing campaign only launches on Marketing campaign only launches on 1/121/12•Users won't get trained before 15/1Users won't get trained before 15/1•Legislation requires us to enable that Legislation requires us to enable that feature on 1/9feature on 1/9

We can't , our ...We can't , our ...•XYZ requires us toXYZ requires us to enableenable that feature that feature on 1/9on 1/9

Deployment: does not mean Enabling Deployment: does not mean Enabling by default ! by default !

Canary , Feature , Dark launches Canary , Feature , Dark launches

•Feature Flags : Feature Flags : ●Only executed if this is on.Only executed if this is on.

•Canary Releases:Canary Releases:●10% of the audience gets a feature10% of the audience gets a feature

•Dark LaunchesDark Launches●Log, do, but don't show the end Log, do, but don't show the end useruser

●Test load of a feature , in real lifeTest load of a feature , in real life

AB testing, Blue GreenAB testing, Blue Green

•AB testing AB testing ●Deploy both alternatives , Deploy both alternatives , ●Show to subsets of usersShow to subsets of users●Compare resultsCompare results

•Blue Green deployments : Blue Green deployments : ●2 identical production platforms2 identical production platforms●Only one is activeOnly one is active● Ideally on the same databasse Ideally on the same databasse backendbackend

Auditors / ComplianceAuditors / Compliance•We do the same, just automatedWe do the same, just automated•Separation of DutiesSeparation of Duties• Man vs Machine Man vs Machine •Authentication and Audit TrailAuthentication and Audit Trail•Full automation, Git logs, Deploy logs, Full automation, Git logs, Deploy logs, no more manual actionsno more manual actions•Have you tried talking to them ?Have you tried talking to them ?

We are already agile !We are already agile !

•We've implemented Scaled Agile We've implemented Scaled Agile FrameworkFramework•We've commited to 4 quarterly releases / We've commited to 4 quarterly releases / yearyear•We can't change thisWe can't change this•Our users can't followOur users can't follow

To ExpensiveTo Expensive•Setting up the stack costs timeSetting up the stack costs time•We don't have the budget to write testsWe don't have the budget to write tests•You also don't have the budget to failYou also don't have the budget to fail•Thats why you are still running Thats why you are still running vulnerable security publically ! vulnerable security publically ! •Operations and development are Operations and development are different budgetsdifferent budgets

•One shot projects , fire and forgetOne shot projects , fire and forget

Culture Hack:Culture Hack:Set up CI / CD for your infrastructure first, Set up CI / CD for your infrastructure first,

If the people running your infra don't know If the people running your infra don't know how CI/CD works , how do you expect how CI/CD works , how do you expect them to support / teach your application them to support / teach your application teams ?teams ?

You also get them to learn about the You also get them to learn about the tooling they will need to support and they tooling they will need to support and they will share the pain and the joy of the will share the pain and the joy of the application developersapplication developers

Culture, Culture,

Automation,Automation,Measurement,Measurement,

SharingSharing

Broken ArchitectureBroken Architecture

•Legacy LanguagesLegacy Languages•Big monolithBig monolith•Stored proceduresStored procedures

•Our team doesn't understand the impact Our team doesn't understand the impact of our changesof our changes

Our application is Our application is •Un buildableUn buildable•Un packageableUn packageable•Un deployableUn deployable•Un configurableUn configurable•Un runnableUn runnable

•Un clusterableUn clusterable•Un scalableUn scalable•Un monitorableUn monitorable•Un measurableUn measurable•Un securedUn secured

““If my If my computer can't computer can't install it, the install it, the installer is installer is broken”broken”Luke Kanies at Luke Kanies at

Fosdem (2007)Fosdem (2007)

As an Ops person As an Ops person ““As a system administrator, I can tell when As a system administrator, I can tell when software vendors hate me. It shows in their software vendors hate me. It shows in their products.”products.”

““DON'T make the administrative interface a DON'T make the administrative interface a GUI. System administrators need a GUI. System administrators need a command-line tool for constructing command-line tool for constructing repeatable processes. Procedures are best repeatable processes. Procedures are best documented by providing commands that documented by providing commands that we can copy and paste from the procedure we can copy and paste from the procedure document to the command line. We cannot document to the command line. We cannot achieve the same repeatability when the achieve the same repeatability when the instructions are: "Checkmark the 3rd and instructions are: "Checkmark the 3rd and 5th options, but not the 2nd option, then 5th options, but not the 2nd option, then click OK." Sysadmins do not want a GUI that click OK." Sysadmins do not want a GUI that requires 25 clicks for each new user.”requires 25 clicks for each new user.”

Thomas A. Limoncelli in ACM Queue December 2010Thomas A. Limoncelli in ACM Queue December 2010

http://queue.acm.org/detail.cfm?id=1921361http://queue.acm.org/detail.cfm?id=1921361

Our process is really complexOur process is really complex

•Different people decideDifferent people decide•Different needs Different needs •Merges are complexMerges are complex•Release management takes agesRelease management takes ages•Testing takes agesTesting takes ages

We don't understand gitWe don't understand git•We've copied our We've copied our svn modelsvn model•We're still in merge We're still in merge hellhell

We still don't understand We still don't understand

•Stop Branching !Stop Branching !•Master only Master only developmentdevelopment•Short lived feature Short lived feature branchesbranches• Short is hours , not Short is hours , not days, certainly not days, certainly not weeksweeks

But our data migrations !But our data migrations !

•FlywayFlyway•DB-MigrateDB-Migrate•Liquibase,Liquibase,

Data flows backwards Data flows backwards

Every small step you take thinking you go Every small step you take thinking you go closer to continuous delivery closer to continuous delivery

From 3 months to 1 monthFrom 3 months to 1 month

From 1 month to bi weeklyFrom 1 month to bi weekly

Makes your delivery process more Makes your delivery process more complexcomplex

Going all the way is much less painfull Going all the way is much less painfull than step by stepthan step by step

ConclusionsConclusions

•Most reasons why you can't are Most reasons why you can't are •WrongWrong•MisconceptionsMisconceptions•On your Backlog alreadyOn your Backlog already•It's a requirement for securityIt's a requirement for security•hapiness(users,developers,ops,managemhapiness(users,developers,ops,management,customers,shareholders,*)++ent,customers,shareholders,*)++

No, No, you can not you can not

not do not do Continuous DeliveryContinuous Delivery

A software project is not done until A software project is not done until

your last enduser is in his grave ! your last enduser is in his grave !

Kris Buytaert, DOD Amsterdam 2013Kris Buytaert, DOD Amsterdam 2013

ContactContactKris Buytaert Kris Buytaert kris.buytaert@inuits.eukris.buytaert@inuits.eu

Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.be/http://www.inuits.be/

Inuits HQInuits HQ

Essensteenweg 31Essensteenweg 31BrasschaatBrasschaatBelgiumBelgium891.514.231891.514.231

+32 475 961221+32 475 961221