View
351
Download
2
Category
Preview:
Citation preview
The Anatomy of Continuous Deployment
At Scale
John Viner (Dev Manager)Mario Visic (Lead Developer of Delivery Engineering)
100 Deploys a Week at Envato
Who is Envato?
A two-sided Marketplace for creative assets that has paid out $400 million to our
“Authors” in the last 10 years
7 Sites where digital professionals can buy web templates, video, audio, code, graphics and 3d
models 7 million registered users10 million items for sale
Themeforest.net
Show me the Tech Stack
Ruby on Rails (100K loc, 25 web servers)
MySql (175 tables)
Memcached + RedisRackspace / VMWare
Amazon AWS
Show me the Numbers!
Peak of 25,000 requests p/min through Rails160m Page Views per Month
500,000 Database calls p/min Serving 200TB of data a month
22,000 Unit Tests, 2,000 Integration Tests
How do we deliver change?
By the numbers … for the month of June 2016
~ 40 developers3000 total builds (2.5 builds p/dev/day)
500 Pull Requests Closed400 deployments (18 p/day avg, 0.5 p/dev/day, peak 25)
4.5 hour cycle time (from first commit to deploy - working hours)
Statistics over Time
Size of a Pull Request (median)37 lines changed3 files changed
Cycle Time of a PR (first commit to deploy)average: 10.8 hours (working hours)median: 2.6 hours (working hours)
Deployments8400 deployments in 2 years
Pretty Charts - Builds
Deployments
Pull Requests
PR Cycle Time (First Commit to Deployment)
Building Envato Market
Feature request
• 2 week sprints• Work with product team to break down work• No task larger than two days• Feature toggle can be used to push incomplete features
Feature request
Implementation
• Write the tests (sometimes first)• Write the code• Peer review• Get it into production
Implementation
Tests live alongside code, stored in Github
Implementation
Github Pull Request are the unit of change, 2 thumbs up by reviewers means the code is good to ship to production
Implementation
Implementation
Buildkite is our CI tool, a merge on master automatically runs a build. A successful build automatically triggers a production
deploy
Deployment
• Entirely Automated• Post deploy monitoring• Quick rollback• Multiple time per hour
Deployment
Samson (open source) is our deployment tool, successful master builds triggers a deploy which is logged in Samson.
Deployment
We can run manual deployments from Samson and view all PRs in a given deployment.
Deployment
We can trace back to specific changes in a deploy.
Deployment
… including the amount of lines changed. A typical PR changes only 37 lines of code across 3 files
Deployment
Why do we do this?
Lowers Risk
Size of Change is Small means …Easier to identify any failureEasier to rollback any failure
Reduces human errors due to reliance on automationSecurity patches can be deployed within minutes
And also
Reduces Waste:Fewer handovers from Development to QA to Ops
Increases Product Agility:New features go to customers sooner to get earlier feedback
Increases developer ownership and engagement:More direct connection between your creation and it being used
How to get there
Start with Culture
Foster an Ownership Culture in Developers
http://webuild.envato.com/expectations/collaboration/4-i-take-responsibility-for-syste.html
Investment in automation of unit tests, functional tests
Tests to cover legacy systems
Add Tests whenever un-tested code is touched
Set Targets and Measure Test Coverage and # Tests
Automate Deployments and Enable Feature Toggles
Automate your manual run sheet one line at a time
Start with Push Button
Feature Toggles help deploy features without showing them
Start Small
Use a less risky or complicated system first
Investment in monitoring and rollback
Knowing that the system is working within seconds becomes critical
Developers need to be “on the pager”
Invest in the ability to Roll Back
Optimise for Mean Time to Recover
Thank you!
Recommended