Continuous Delivery at Turner Broadcasting System, INC

Daniel WesterChief Engineer, Turner Broadcasting System, INC

2

Built it, Test it, Ship it

Who we are

3

and more…

Continuous Delivery Components

4

Continuous Integration

5

Continuous Integration

6

7

“What makes continuous deployment special is deploying every change that passes the automated tests (and optionally a short QA gate) to production. Continuous deployment is the practice of releasing every good build to users – a more accurate name might have been “continuous release”.”

From: Continuous Delivery vs Continuous Deployment by Jez Humble http://continuousdelivery.com/2010/08/continuous-delivery-vs-continuous-deployment/Copyright © 2011 Jez Humble - All Rights Reserved. This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License (http://creativecommons.org/licenses/by-sa/3.0/us/ )

Continuous Deployment

8

“Implementing continuous delivery means making sure your software is always production ready throughout its entire lifecycle – that any build could potentially be released to users at the touch of a button using a fully automated process in a matter of seconds or minutes.”

From: Continuous Delivery vs Continuous Deployment by Jez Humble http://continuousdelivery.com/2010/08/continuous-delivery-vs-continuous-deployment/Copyright © 2011 Jez Humble - All Rights Reserved. This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License (http://creativecommons.org/licenses/by-sa/3.0/us/ )

Continuous Delivery

Continuous Delivery: Deployment

9

Configuration Management

10

Notifications

11

Continuous Delivery

12

Our journey starts

13

•CNN.com created in 1995•Everything manual•Dev environments shared•A lot of development being done in production

•No configuration management

Conversation

14

Customer

Developer

Hey! This feature no longer works! It worked 10 minutes

ago. Oops! That was me. Fixing it

now.

Wild West Development

15

Shell script deployments

16

• Set of shell scripts monitoring a directory handles deployments.

• UI created in front of folder to allow developers select files to promote to an environment.

Conversations

17

Dev Lead Developer

Dev Lead Developer

The Customer reported that this just broke – who changed

something?

It wasn’t me – the change I just

pushed out was for this other thing

Did you push out the new configs as

well?Oh…

Continuous Delivery

18

Partial CI Builds

19

•Replaced shell scripts with Ant and CI server

•UI allow Developers select files•Files tagged in SCM with environment and date

•Backend managed deployment to environments

Demo

20

Controlled single change deployment

21

Continuous Delivery

22

Challenges with partial builds

23

•Environments are not re-creatable•Not scalable as development team grows

•Does not scale for large changes

CI with manual deployment

24

•Web Applications•Compiled applications•Manual deployments done through Configuration Management tools

Continuous Delivery

25

Challenges with CI/manual

26

•Files are passed around•Manually maintained•Builds are incremental, but release is not

•Prone to environmental inconsistencies•Intra-project differences

CNN.com Elections

27

•AJAX based application•Iteration developed•One build plan per environment•Deployment to production - manual

Deployments through cloning

28

Conversations

29

Dev Lead Developer

Dev Lead

QA

I thought that you said that the fancy feature would be in

the build?

Done

Oh! I forgot to merge it in time.

Can you do another build?

The new build that was just delivered

broke a lot of functionality.

Continuous Delivery

30

Integration with CM frameworks

31

• Integrated with CM for external projects

• Infrastructure driven

Not just for SCM based SW

32

Continuous Delivery

33

Continuous Delivery

34

Continuous Delivery

35

• “A new” web development platform brought in

•Requirement of “zero-touch” of production

•Single build for all environments.

Continuous Delivery

36

Continuous Delivery Iteration 2

37

•Monitoring system integration•Build promotion history•Remediation can occur on Plan or job

Demo

38

Why integrate build sys and Release Mgmt?

39

•Source Code Correlation•Guaranteed no manual steps•For the developers – one tool

Conversations

40

Dev Lead

Developers

Dev Lead QA/Customer

I see you have changes lined up– are they ready to go into

the staging env?

Build 232 is in staging

Yes.

Build 232

looks good.

Dev Lead

Build 232

going out to prod

What the NOC sees

41

What the QA team sees

42

What the Developer sees

43

What the Infrastructure team sees

44

•Concentrate on automation•Developer accountability for production increases

•Less firefighting•Next project…

What Management sees

45

•NOC Awareness•Unit and Functional tests•QA Tested•Automatic deployments with rollback capability

•Speed to market with minimal delays•Efficiency in the process•Removal of artificial roadblocks

Continuous Delivery

46

Challenges

47

•Full build concerns from various parties•Development and Infrastructure teams must be in sync

•Use SCM, Build and CM systems for their purposes

•Initial investment cost for new project types•One artifact to rule all environments

Impact

48

•Delivery time to market - down•Quality of deliverables - up•Cultural change (more collaboration)

•Change tracking on source changes

49

Continuous delivery requires sharing operational ownership between dev & ops, and lets us rapidly develop sites in parallel

#summit11