Embed Size (px)
DESCRIPTION
In today's digital economy, it is not the big that eat the small, but the fast that eat the slow. Traditional IT project management has led to spectacular failures in terms of delays and risk management, on the other hand web leaders like Amazon, Netflix or Google have reached incredible velocity through disruptive implementation of agile principles. Among the different agile movements, Devops gathers development and operation experts on how agility can be implemented all the way from the technical conception to the production environment to reach maximum velocity. Learn from our experience on many large-scale Symfony2 projects, with up to 15 developers. I will present what Devops means for us and how it helps us deliver much faster and much more reliably. We will see the many cultural, management and technical challenges we faced and the solutions we tried, based on a very powerful technical stack: Symfony2, OpenStack, Puppet, Vagrant, Capifony, Jenkins, Behat and many others.
Citation preview
THEODO8/11/2013 1
How to accelerate your IT flowThe Devops Wonder
THEODO8/11/2013 2
How to accelerate your IT flowThe Devops Wonder
A preach in the Symfony Live Berlin church on finding the path to fast IT
THEODO8/11/2013 3
The need for speed in IT
The Devops wonder… and facts that prove it
How to start practising Devops, step-by-step
THEODO8/11/2013 4
Business needs velocityThe need for agility
THEODO8/11/2013 5
V-Cycle is slow and risky
Source: http://www.ucl.ac.uk/syseng/images/vDiag
?The problems of the alternative
THEODO8/11/2013 6
Much slower and riskier than what can be estimatedThe natural mistake
A study by two Oxford professors on 1 471 large IT projects shows :• 1 project out of 6 has cost on average 3x more than expected!• large-scale computer spendings were 20 times more likely to spiral
out of control than expected!
Oxford study: http://users.ox.ac.uk/~mast2876/WP_2011_08_15.pdfFox-Meyer bankruptcy: http://fr.slideshare.net/jmramireza/the-foxmeyer-drugs-bankruptcy-was-it-a-failure-of-erp-2332065
Large-scale example:• FoxMeyer Drugs’ bankruptcy after switching
brutally to SAP
THEODO8/11/2013 7
Our brain sees the world through Bell curvesBlack swan blindness
Average scenario
Good case scenarioWorst case scenario
Probability density of possible outcomes
THEODO8/11/2013 8
Complex systems do not act like Bell curvesBlack swan blindness
Average scenario
Good case scenarioWorst case scenario
Probability density of possible outcomes
THEODO8/11/2013 9
This is why small iterations are so important!Lean IT
Machiavel: Divide et impera! (Divide and conquer)
THEODO8/11/2013 10
Agile development is about small FAKE iterations Agile development
Source: http://www.mypmhome.com/scrum-methodology/
THEODO8/11/2013 11
Because features will just pile up in front of IT operationsThe dev VS ops silos
THEODO8/11/2013 12
The Scrum prophets showed the light, and the first steps to itScrum is not the end
THEODO8/11/2013 13
The need for speed in IT
The Devops wonder… and facts that prove it
How to start practising Devops, step-by-step
THEODO8/11/2013 14
Devops is about solving the silos problemThe definition of Devops
Devops is• making the whole IT organisation work as a unique team• with a common goal: business-oriented IT performance
THEODO8/11/2013 15
The Devops trinity: tools, process, peopleThe three layers of Devops
Devops is• Tools: how to do it• Process: when to do it• People: why to do it
TOOLS
PROCESS
PEOPLE
DEVOPS
THEODO8/11/2013 16
The people part is important: devs and ops are different!Devops is a lot about culture
DHH, the epitom of the hipster developer
Stallman, the epitom of the paranoid GNU/Linux guru
THEODO8/11/2013 17
Devs want speed and think short-termDevops is a lot about culture
The typical dev The typical ops
I just coded this really cool feature using
CouchDB/PouchDB/Node.js/S3
Let me deploy it now, business wants it online
today!
THEODO8/11/2013 18
Ops want stability and think long-termDevops is a lot about culture
The typical dev The typical ops
CouchDB/PouchDB/
Node.js/S3???
THEODO8/11/2013 19
In the worst case, you get neither short nor long-term resultsDevops is a lot about culture
The typical dev The typical ops
What about Backups?
Monitoring?
Performances?
Security?
Documentation?
Scalability?Business is not going to be happy…
THEODO8/11/2013 20
Devops efficiency is highly documented on the webDevops studies
• IT Ops & DevOps productivity report 2013 by RebelLabs surveyed 620 engineers
• 2013 State of DevOps Report by PuppetLabs & IT Revolution Press surveyed 4000 IT ops and devs
• And all the testimonials from web leaders like Amazon, Netflix, Etsy, Flickr, etc. (at Velocity Conference, DevopsDays, etc.)
THEODO8/11/2013 21
Devops ship code 30x more frequently!2013 State of Devops report by PuppetLabs and IT Revolution Press
Mean time between deploys
THEODO8/11/2013 22
Devops ship code 8000x faster!2013 State of Devops report by PuppetLabs and IT Revolution Press
Mean time spent deploying
THEODO8/11/2013 23
Devops have 50% less failures!2013 State of Devops report by PuppetLabs and IT Revolution Press
Failed deployments rate
THEODO8/11/2013 24
Devops restore service 12x faster!2013 State of Devops report by PuppetLabs and IT Revolution Press
Mean time to resolution
THEODO8/11/2013 25
Amazon deploys in average more than 300 times an hour!Devops success stories
• Founded in 1994, 61 billion$ revenue in 2012• Mean time between deployments : 11.6s (310 times per hour)• Max number of deployments in an hour : 1,079• Mean number of hosts receiving a deployment : 10,000
http://assets.en.oreilly.com/1/event/60/Velocity%20Culture%20Presentation.pdfhttp://en.wikipedia.org/wiki/Amazon.com
THEODO8/11/2013 26
At Etsy every engineer deploys, and starts on the first day!Devops success stories
• Founded in 2005, 1 billion$ transactions in 2013• 250+ committers, everyone deploys• 30+ deploys a day• Every engineer at Etsy deploys on the first day
http://codeascraft.com/2012/03/13/making-it-virtually-easy-to-deploy-on-day-one/https://speakerdeck.com/astanway/bring-the-noise-continuously-deploying-under-a-hailstorm-of-metricshttp://gigaom.com/2013/08/23/meet-the-man-behind-new-yorks-other-billion-dollar-internet-company-this-one-makes-money/
THEODO8/11/2013 27
The English government did a radical adoption of Devops in less than a year!Devops success stories
What is gov.uk? “The websites of all government departments and many other agencies and public bodies”• Alpha launched in 2011 with 4 devs. Site was live in october 2012• 15-20 deploys a day• Main selling point: faster than other suppliers.
• “this is what we’d like. Can you work out if it’s possible? let’s meet again in a month”• -> built, shipped in 3 days
http://vimeo.com/album/2384821/video/66622266https://github.com/philandstuff/devopsdaysparis#kushal-pisavadia-kushalp-how-we-ship-software-at-govukhttps://www.gov.uk/service-manual
THEODO8/11/2013 28
The need for speed in IT
The Devops wonder… and facts that prove it
How to start practising Devops, step-by-step
THEODO8/11/2013 29
Creating a culture of teamwork between devs and opsThe three (or four) maturity levels of Devops adoption
1. Bring devs closer to the constraints of operations
2. Bring ops to empower devs
3. Further embed devs and ops in each other’s teams
THEODO8/11/2013 30
Let us start by teaching junior devs basic ops conceptsThe common developer background
The typical junior developer• learnt Java at University• uses Windows… because of the games • has installed Linux because he is curious• maintained a website for one of his clubs
at University…• … « deployed » changes using FileZilla
Source: http://www.essentialbaby.com.au/
THEODO8/11/2013 31
Step 0: make the baby dev a respectable devBasic dev skills
• Linux development environment• Git versioning• Correct git branching strategy• Scrum methodology• Unit and functional tests
Source: http://www.aceshowbiz.com/
THEODO8/11/2013 32
Step 1: give the devs a server to play withOne dedicated server per developer
• Give a tiny instance to everyone• Free to do whatever you want,
whenever you want• …as long as it is legal…
«I improved a lot when I started renting my own server»
THEODO8/11/2013 33
Step 2: make them set up a simple server… at least twiceBasic system administration skills
• Amazing how many junior devs have never set up a server…• Twice: to get the taste for automated provisioning
What we do:• First a simple setup with Apache + mod_php• Then a more subtle setup with Nginx + fpm
THEODO8/11/2013 34
Step 3: bring devs to deploy… and understand how it worksAutomated deployment
• Shell scripting: forget it.• Capistrano: too magic?• Fabric: good compromise?
Our experience: every project has a deploy.py script in a «devops» folder (or a Capfile…):• Everybody is now able to deploy automatically• Juniors seem to check if there is an experienced guy around
before deploying…
THEODO8/11/2013 35
Fabric makes deployment automatic and not too abstractFabric sample
@roles('prod')def deploy(): tag = "%s/%s" % (_getrole(), strftime("%Y/%m-%d-%H-%M-%S")) local('git tag -a %s -m "%s"' % (tag, _getrole())) local('git push --tags')
run('git fetch’) run('git fetch origin --tags') run('git checkout ' + tag)
THEODO8/11/2013 36
Step 4: make monitoring an obvious stepMonitoring
• Monitoring has become very easy: think NewRelic and AppDynamics• By making it an early step, you positively change the dev culture• Easy access to: performance issues and hidden 500 errors
THEODO8/11/2013 37
Step 5: make continuous integration a dev responsibilityContinuous integration
• Setting up Jenkins is easy: all devs should be expert at doing it
• To avoid conflicts between jobs: use a separate slave node for each projet
• Why not Travis-ci? Good solution, but Jenkins proficiency is necessary
https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds
THEODO8/11/2013 38
Step 6: normalised environment with VagrantDevelopment VM
• Macbook, Ubuntu, 32 bits, memory_limit=2048M, ulimit…
• Development environments are different from production in subtle ways
• Vagrant: wrap a VirtualBox VM that launches the same OS flavour
• Bonus: « vagrant package » and you can share your box with a new developer, makes him operational in a few minutes
• Important performance tips: http://www.whitewashing.de/2013/08/19/speedup_symfony2_on_vagrant_boxes.html
THEODO8/11/2013 39
It is time for ops to collaborate and empower devsProduction feedback into development
THEODO8/11/2013 40
Step 7: use Vagrant with Puppet/ChefAutomated provisioning
• Puppet (or Chef) is painful for a dev. But by now he should find manual provisioning painful too!
• Make an experienced ops create a Puppet/Chef template• Make an experienced dev challenge everything and simplify things!
?
THEODO8/11/2013 41
Step 8: enable self-serviceInfrastructure as a Service
• Public cloud or private cloud• OpenStack is now mature (enough)
THEODO8/11/2013 42
Step 9: make backuping easyBackups
• S3 / Swift to push backups to• Or solutions like Idera ServerBackup
THEODO8/11/2013 43
Step 10: create a self-service load testing platformPerformance
• Create an easy to access (VNC) server with a Jmeter and large bandwidth
• Let the ops teach the devs what a good lead testing scenario is• Test the load regularly (if necessary) with devs+ops together
THEODO8/11/2013 44
Step 11: enforce pair-devopsingPair devopsing
• The dev team has an issue with Puppet/Chef?• The load tests indicate a subtle performance issue?• Ops see cryptic errors in the logs?
Pair devopsing!
• Scrum compatible• Works with curious people
hire curious people
Source: http://www.turnbacktogod.com
THEODO8/11/2013 45
Step 12: use visual management as a common incentiveVisual management techniques
Example: how do you make devs focus on performance?Þ First make it part of the DONE definitionÞ Include performance solutions in the standard provisioningÞ AND do visible performance graphs
THEODO8/11/2013 46
How do you bring devs to feel responsible for production?Sharing responsibility
Why do ops feel that responsible, and not devs?
Dev: «It is not my job»
… it is up to the business to decide if it is worth putting devs on call.
THEODO8/11/2013 47
Next steps: learn from the web leadersSources
• Velocity conferences• http://lanyrd.com/2013/velocity/coverage/• http://lanyrd.com/2012/velocity/coverage/• Etc.
• Gene Kim’s “Top 11 things you need to know”• http
://www.thinkhdi.com/~/media/HDICorp/Files/White-Papers/whtppr-1112-devops-kim.pdf
• Etsy’s blog: http://codeascraft.com/• Netflix’s blog: http://techblog.netflix.com/
THEODO8/11/2013 48
Iterate to improve continuouslyContinuous improvement
Scrum, Devops, Lean Startup: it is all about continuous improvement!
• Measure everything• Identify problems• Try countermeasures
And iterate
THEODO8/11/2013 49
And I hope that you now believe in the Devops wonderThe miraculous performances of Amazon
Werner Vogels, CTO at Amazon, 300+ deploys per hour
