Making the Agile Leap to Continuous Deployment

Making the Agile Leap to Continuous Deployment

Ethan Ram / 15 Feb 2014V1.1

© All rights reservedWant to use slides from this presentation? Ask for permission: [email protected]

Ethan Ram - Bio• Chief architect of Videobet• Leading Videobet’s R&D and OPs

teams Agility efforts• As VP R&D of GameGround

transformed the R&D to run in Continuous Deployment environment

• Hacker in soul and a Windows Internals hobbyist

Find me here: • [email protected]• LinkedIn• The Null Terminator

blog

2/15/2014 © All rights reserved 2

Videobet• A Playtech division • Innovative Gaming

Platform Provider– Established 9 years ago– 150 employees in Tallinn

and Kiev– Operate in 9 countries– 60% of UK market

• R&D– Front-end teams (C++)– Server teams (Java)– Game development teams


• Why Continuous Deployment• Setting the goals • Deployment technics• Development technics• Making the leap• Some unexpected results • Q&A

Agenda


Agile Manifesto Principle #1

Our highest priority is to satisfy

the customer through early and

continuous delivery of valuable

software.

At SlideShare, instead of big launches, we release code all the time... it reduces risk.

Since we are putting out small changes at a time, if it breaks we know what it is and can

roll back.Rashmi

We have been doing continuous deployment at Plex since our inception in 1995. We

formalized the process with our own “deploy” button in 2003, and hit the 25k deployment

mark less than two years thereafter. We currently average between 50-100

deployments per day to our massive online ERP system.Jerry Foster

Goal: Continuous Deployment

2/15/2014 © All rights reserved

WordPress.com - we’ve averaged about 16 product releases a day, every day for the last

four and a half years… Everyone in our company has access to a deploy button that releases the latest checked in code to about 400 production servers in our web tier in less

than 30 seconds (across 3 data centers). Toni Schneider

One of the exciting things… about Digg is continuous deployment - when developers fix a bug or add a new feature, there's no need

to wait for a scheduled release… the turnaround time for a change drops

dramatically.Andrew Bayer

5

Wix “From Jan ‘13 to Jun ’13 we did 1500 Deployments, 470 A/B Tests,

200 Feature Toggles”Yoav Avrahami

Flickr "10+ Deploys Per Day" Steve Conover


Major ReleaseMajor Release

No. I Can’t Do That • “We’re starting this campaign next week and

I need the product to do X”• “We need to stop showing this feature to

non-US customers”• “I think this idea would work. When can we

start A/B testing live?”• “I need to know if users are clicking on this

link and if they see this error message”• “Forget about the other features. We have to

go live by the end of the month!”



Continuous Deployment: Continuous Deployment: Yes We Can!Yes We Can!

8

How to Streamline Deployment?

• Speed-up Dev-ops• Lower risk of deploying something broken• Streamline feature implementation


???

Monitoring & Rollback• Must haves – real-time– Monitor that features are working,

not just servers– Monitor the business results– Visualize and put it on the big

screen– One-click rollback

• Rollback: the ultimate risk mitigation– Prepare to fail!

• Will the rollback work?– Rollback drills


Deployment & Servers SetupGo from – 8 hrs downtime on upgrade• Physical machines• IP based network topology• Layered servers: web-apps,

backend, cache, db• Feature servers• Each team has a different

deployment setup and scripts• Complex database schema upgrades

To – 8 mins deploy, no downtime• Virtual machines• DNS based network topology• All servers on each VM

– DB too!!!• Same VM setup on all

environments – With different config– With different data-set

• One deployment script run on dev machine >> production– Target tag and environment name

as input• Versioned mini schema

migrations


IT’S EASY!

DB Versioning With Flyway DB• Mini schema migrations kept with source

code– Applying migrations added between v1 to v2 is

easy– Roll-back of migrations supported built-in

• Each developer has her own DB • Merging team-branch migrations is even

simpler than merging code• Integrative with Maven/Ant/scripts and Java

APIs– Server can upgrade/lower DB schema by itself

• Data for unit-testing can be added and removed on the fly


Flyway DB makes

you Agile!

Staging on Production• Get real users to use new version ASAP– Call it Beta, Early Availability, Version Trial, Test-

bed

• Deploy new version to part of the servers and direct it traffic of part of the user-base– Deploy to a subset of servers– Run manual tests & automated tests– Direct some traffic to new version– Monitor! Measure!– Deploy to the rest after some time

• Not only for consumer-based internet products!


• Integration Test Automation– Focus first on covering main data flows–More main features, less details

• Part of the Continuous Integration– Build > Unit-test > Integration Tests– All tests must pass to allow build promotion

• Run the same tests on all environments– Developers’ & production environments too

• Don’t give-up on manual testing• ‘Test frenzy’ as version signs-off– One hour session, structured, lead by QA– ALL employees join. The secretary too.

Quality Assurance


…not so easy…


BACKLOG SELECTED IN PROGRESS RESOLVED VERIFIED TESTBED DONEBacklog

• Bugs• Improvements.• Daily

prioritization

Selected

•Feature owner assigned•Task Breakdown•User stories•Design/gfx•Test plan

In Progress

• Coding• Unit testing• Integration

testing

Resolved

• Need verification

• Drop build to QA

• Automated integration testing

• Manual testing

Verified

• Verified• Ready to be

installed on TestBed

Testbed

• Optional step: install on TestBed

• Regression Testing

Done

• Running on production

• Done / Not relevant

R&D Kanban Board: CD Workflow

• Enforce WIP limits• One owner for each column• Feature owner responsible for feature end-to-end…

• QA plan and test completion (no need for bugs!)• Unit testing and integration testing• Use DBA/HTML/FLASH and QA resources as needed• Test it on testbed/production

16

Developing on Trunk


Use feature toggle flag Use feature toggle flag 2/15/2014 © All rights reserved 18

• Use feature toggle flags• Each dev has a full system - update, compile, run it all, test-it• Write backwards compatible code

– Assume some servers/services do not have the new feature/code yet

– DB fields may be missing– Mark older APIs with [deprecated/obsolete]

• Dynamic Configuration– Central config, dynamically loaded– Plan for no app/system restarts

• Schema changes are added to code as DB versioning scripts• Larger features that require a branch are rare!• Education: Do not break the trunk!

Developing on the Trunk


Thorn Free?!?

Product: From Revolution to Evolution


V2.2

V2.3 / R403

R417

Leaping into CD• Initiative must come from high mgmt.

– It’s a large investment… and risky• It’s not all or nothing >>>

– First milestone: weekly release– Set a deadline and commit: “we start

releasing once a week right after the next major release in September”

• It’s a project to deliver– Multi-function task-force: devs, QA,

deployment– Full time!– Plan a lot – detailed task list etc.

• Training to all teams: process, tools• Decide on a CD weekly schedule

– Tuesday afternoon is best time for a release• Make results visible early!

– And celebrate success!


money time

The first CD release felt a bit like this…The first CD release felt a bit like this…..Just Go For It!..Just Go For It!


Some Unexpected Results• … the sky did not fall on us • Excellent process transparency

– R&D stopped fighting with Product Mgmt.– Team leaders stopped blaming each other– Devs thought it’s much more fun– Everyone got more involved

• We managed to attract much better engineers• R&D managers thought its awesome!


Q&A

Want to use slides from this presentation? Ask for permission: [email protected]


Thanks!

Thanks for contributing!Yoav Avrahami @ WixMike Mcgarr @ Excella

Technology

Making the Agile Leap to Continuous Deployment