1

DevopsDays Tokyo 2012

Friday, May 25, 12

Acknowledgments

This presentation builds on the work of many. DevOps (and cloud operating model) thought leaders, including Damon Edwards, John Allspaw, Jez Humble, Adam Jacob, Luke Kaines, Jesse Robbins, Chris Read, Andrew Clay Shafer Gene Kim, Patrick Debois and John Vincent. Special thanks to Cameron Haight of Gartner for collaborating and codifying many of the Devops community ideas.

2

Friday, May 25, 12

What is Devops?

3

Friday, May 25, 12

Success Stories

Instagram• In 2 Years

30 Million users2 Backend engineers (devops)1 Billion dollar acquisition by Facebook

Dropbox• Last 3 years

+50 Million new users$240 million per year in revenue

Etsy• Last 3 Years

Hit 1 Billion pageviewsHosting 875,000 sellers and 15 million users worldwide$525 million in sales in 2011

4

Friday, May 25, 12

Raising the Bar for IT Infrastructure

Netflix• 100% cloud• Developer focused PaaS• Revenue Growth: 23% (three average)• Profit Growth: 48% (three average)

Amazon• Production deployment every 11.6 seconds (weekday) • 1,079 deployments in one hour (record)• ~0.001% of deployments actually cause an outage

5

Friday, May 25, 12

What do they have in common?

Kaizen• Continuous improvement

Muda• Eliminate Waste

Mura• Unevenness - (reduce inventory)

Muri• Overburdened (standardization)

6

Friday, May 25, 12

Now in English

Optimize for minimal operation burdenInstrument simplicity for the business Instrument everythingOptimize for Happiness

7

Friday, May 25, 12

Common Technical Characteristics

Cloud or Cloud like infrastructureOpen Source and Open CultureThey practice a form of Devops/Agile

8

Friday, May 25, 12

So What is Devops?

Improving collaboration and communication between development and operations teams.An adoption of methodologies (and some technologies) used in software development by operations teams.A renaissance of infrastructure technologies that is strongly influenced by cloud computing.

9

Cameron Haight - Devops Workshop

Friday, May 25, 12

What do we say?

CLAMS

• Culture• Lean• Automation• Measurement • Sharing

10

subscribe in

iTunes

Friday, May 25, 12

Simply Put

Improving the business value of the work done

within IT.

11

Text

Cameron Haight - Devops Workshop

Friday, May 25, 12

Characterization of Devops

12

Ah-ha! Ka-ching!Wall o

f C

onfu

sio

n

OpsDev

Friday, May 25, 12

Characterization of Devops

13

Ah-ha! Ka-ching!

OpsDev

Wall of ion

onfusC

Remov

e

Shorten

Friday, May 25, 12

The Convergence of Devops

Devops History

• Agile Infrastructure Thread• Velocity Thread• Lean Startup Thread

The Convergence of Devops

http://bit.ly/itrev2

14

Friday, May 25, 12

Agile Infrastructure Thread

2005

2008

2009Friday, May 25, 12

Velocity Thread

2006 - 2008

2005

2009

2010

Friday, May 25, 12

Lean Startup Thread

Customer Development Model

Friday, May 25, 12

The Seven Sins of Waste (Applied to Devops)

Transportation• Reduce boundary hand-offs/approvals

Inventory• Leverage cloud for JIT (just-in-time) resource

Motion• Automate and implement self services

Waiting• Manage flow (Visualize, WIP limits) / Kanban

Over-processing• Implement MVP / min. viable product

Overproduction• Optimize for the whole / Theory of Constraints

Defects• Bring the pain forward / 5 whys18

Friday, May 25, 12

Culture Principles

No rock star mentality• Shared contributions

Healthy attitudes towards failure• Failures are leaning opportunities

The problem is the enemy• No blame games

No victims• Shared blame

Develop shared metrics• Focus on end goal

Alignment of Purpose• Shared goals/ Slay the Dragon

19

Friday, May 25, 12

In Development

Done means released• Code deploy not code complete

Infrastructure as code• Configuration is code and needs control

Version control everything• Everything is an artifact (scripts,xml, source)

Frequent releases• Self service / Continuous delivery

Test end to end• Test driven code and infrastructure

Instrument operations • Feature flags / Canary releases / Immune systems

20

Friday, May 25, 12

In Operations

Automate where possible• Desired state consistency

Instrument pervasively• Collect data to detect trends early

If anything fails stop the line• Reduce technical debt early

If it’s hard do it more often• Practice make perfect

Enable graceful degradation• Some availability is better than none

Reprovision not repair• It’s easier to recover to a known state

MTTR vs MTBF21

Friday, May 25, 12

The Organization

Hack Days• Creating collaborative projects

Free Time• Allowing special free project time

Embedded Engineers• Dev in ops or ops in dev

Chat Rooms• Skype, Hipcat, Watercooler

Fun Working Environments• Games, reading rooms, bars

22

Friday, May 25, 12

Technical Debt

23

Friday, May 25, 12

Technical Debt - Vicious Cycle

24

Technical Debt

Vicious Cycle

ToxicOperations Terminal

Friday, May 25, 12

Tall of Two Startups

25

Friday, May 25, 12

Meat to Math Ratio

26

Friday, May 25, 12

A Cloud Gone Wrong

27

Friday, May 25, 12

Clouds Gone Wrong

Friday, May 25, 12

Friday, May 25, 12

“As-is”

app devs

middlewaredevs

operations

systemeng

EC2

servertemplates

yum repo

RS deploymentRS

deployment

puppet

S3

code,content

code

TARTAR

TARTAR

platform

TARRPM

TARRPM

servertemplates

EXE

AUTO

TOOLS

Provision-time builds

CONTROL

restarts

deploys

reconfigs

PROVISION RELEASE

Friday, May 25, 12

Friday, May 25, 12

pup

pet

SVN

active

directory

splunk

instance create

resource model

users packages

new node Node a

commands

right

scale

run

deck

hud

son

yum

repo

nag

ios

packages

sys cfg

eventslog data

agent cfg

CONTROL PROVISION RELEASE

Everything starts here

Fully Automated Infrastructure

Friday, May 25, 12

Acknowldegements

This presentation builds on the work of many. DevOps (and cloud operating model) thought leaders, including John Allspaw, Jez Humble, Adam Jacob, Luke Kaines, Jesse Robbins, Chris Read, Andrew Clay Shafer Gene Kim, Patrick Debois and John Vincent. Special thanks to Cameron Haight of Gartner for codifying many of my and other’s ideas in a grown up way.

33

Friday, May 25, 12

@RealGeneKim, genek@realgenekim.me

The Prescriptive DevOps

• Capture and codify how to start and finish successful DevOps transformations– Create isomorphic mapping between

plant floors and IT shops– Co-authoring with Patrick DeBois,

Mike Orzen, John Willis– Describe in detail how to replicate

the transformations describe in “When IT Fails: The Novel”

• Goals– How does Development, IT

Operations and Infosec become dependable partners

– How do they work together to solve business problems (and Infosec, too)

Friday, May 25, 12

Resources

35

DevOps Cookbook

Friday, May 25, 12