CD@ yahoo 2015 iThome devops conf

Continuous Delivery @ Yahoo Baiyi Ying⎪ Sep 01, 2015

Questions

Version Control

NASA's Marshall Space Flight Center@flickrcc

Code Change to Production

How Frequent ?

Deliver ?

Yuri Samoilov@flickrcc

Deliver ?


VALUE

照顧花園...

優雅吃飯

久坐不累

Katerina Kamprani - The Uncomfortable project All rights reserved

nisha@flickrcc

visualplun.ch@flickrcc

照顧花園...

優雅吃飯

久坐不累



nisha@flickrcc


照顧花園...

優雅吃飯

久坐不累




nisha@flickrcc


照顧花園...

優雅吃飯

久坐不累




花惹發?

Arturo J. Paniagua@flickrcc

nisha@flickrcc



Deliver ?


things customers want to use

How do you know you’re delivering values?

IDEAS

CODEDATA

build

lear

n

measure

VALUE


WASTE visualplun.ch@flickrcc

VALUE


WASTE visualplun.ch@flickr visualplun.ch@flickrcc

In the Good(?) Old Days…..

Development

Testing

Testing

Release Go/No-go

1 week before D day

Release Go/No-go

Release Day

Release Day

wolfgangfoto@flickr

Development Testing

Release Go/No-go

Release Day

wolfgangfoto@flickr

Why Change ?

VALUE


WASTE visualplun.ch@flickr

Speed

Small Batch

Easy toIdentifyIssues

Check&

Revise

Quality

Build the Right Thing

Build the Thing Right

Good Working Code

Lots Testing

Lots Auto Testing

Eliminate Human

Involvement

Continuous Delivery

From District 9 movie

COMMIT TO PRODUCTIONWITHOUT HUMAN INTERVENTION

Results

CD Adoption Rate

Months

Multiplier

Production Pushes

APAC Production Pushes

PO: 好開心!!設計的 features可以很快的上 production 得到 users 的反應

SE: release 成本變小，incidents 沒有變多

Dev: 開始習慣先寫測試。測試變得更重要了,因為如果沒測好，就直奔production了

QA: Incremental changes rock!

Dan Goodwin@flickrcc

IDEAS

CODEDATA

build

lear

nmeasure

Lessons Learned

CD is Simple

Version Control


Certify

Build

Build

Certify

Version Control


Certify

Build

Build

Certify

Version Control


Certify

Build

Build

Certify

Version Control


Certify

Build

Build

Certify

Not Cheap Nor Easy

We Thought

William Warby@flickrcc

In Reality ….

Tools v.s. People

sean@flickrcc

hakanforss.wordpress.com

● tools will never be perfect ● can implement CD with imperfect tools


● Mindset Change● Culture Change● CD is the Norm● All-in

How

Executive Mandate

Lost Planet Airmen (1951)

NO CD, NO LAUNCH, NO FOOLING

sean@flickrcc Nick Royer@flickrcc

http://www.imdb.com/title/tt0043758/



5 Stages of Adopting CD

5 Stage of Grief

Denial


AngerDavid K@flickrcc

Bargaining

mark engelbrecht@flickrcc

Depression

marc latermouille@flickrcc

Acceptance

Denial


AngerDavid K@flickrcc

Bargaining

mark engelbrecht@flickrcc

Depression


Acceptance

5 Mental Stages of Adopting CD

Denial 否認

蛤？難道你的老闆的老闆的老闆．．． x N不是 Marrisa 嗎？

全面CD ? 怎麼可能!應該不包括我們吧？

別鬧了!!Projects 都作不完了，一堆 features 等著要上...都這麼忙了，還來搞這個!!

Anger 憤怒


Bargaining 討價還價

好啦好啦，我們會作啦!不過手上這個 project 超重要，沒有人力時間弄 CD，可不可以幫這個 project 申請 exception ?

David K@flickrcc

持續洗腦

Depression 沮喪

kris gabbard@flickrcc

● fear● misunderstanding● doubt

Version Control


Certify

Build

Build

Certify

uncle bob@Ruining your Test Automation Strategy.

Nate Steiner@flickrcc

天啊! 要怎麼把這 30K 隻manual tests 改成自動化測試程式？

Well, you don’t

Acceptance 接受

● No manual testings● No manual commands in production● Not just an engineering project


Aggressive Audacious Goals by Phases

Git in 30

1800 repos

30 D

Build Farms in 30

130 Farms 4 Centralized Farms

30 D

40K Jobs65K Builds / days

CD in 2

Scale & Diversity

cellanr@flickrcc

Single Way

Common Terminology

Common Principles

Tools & Services

Teams to Figure Out

Common Terminology

■ Package● compiled software, scripts, configuration or other data● smallest granularity of deployment● versioned

Common Terminology

■ Image● a set of packages● determines the functionality a

node will have

■ Component● runtime entity● defined by an image

Common Terminology

■ System● one or more components ● certified, launched, and operated together.

Common Terminology

Principles

■ Repeatable & Reproducible ● Versioned● Immutable Artifacts

■ Traceable● Artifact -> all the changes made

■ Version Control● source code, launch scripts and configurations, test code, test data.

Version Control

Package Component System

Package

Certify

ArtifactStorages

Build Build

PackagePackagePackage

Image

Deploy

Certify

Tagger

Build

Prod

PC1

PPC2

P

Manifest

DeployCertify


ArtifactStorages

Tools & Services■ Build & Workflow

● Jenkins ■ Artifact Storages

● Proprietary ● Artifactory

■ Tagger● Proprietary

■ Visualizer● Proprietary

■ Software Delivery ● Proprietary● Chef

■ End-2-End examples■ STOP development and Do it■ Use Senior Engineers■ Pipeline Demos and Reviews

Other Tactics

Results Recap

■ 7X More Production Pushes ■ Quality

Test

able

Cod

e

Ref

acto

ring

CD

Dec

oupl

ed

Des

ign

Prin

cipl

es

Dat

a D

riven

TDD

BD

D

Ope

ratio

nal

■ First Domino ● Product Excellence● Engineering Excellence

……

…..

etc

After Adopting CD

Before Adopting CD

回不去了!誰想回去啊？

Summary

visualplun.ch@flickr

Jesse! S?@flickrcc

Nathan E Photography@flickrcc

sean@flickrcc
