DevOps or: How I Learned to Stop Worrying and Love the Cloud

o r :How IL e a r n e dto S to pWorrying

a n dLove the

Cloud

DevOps@nxhack 2010/12/31

INTRODUCTION TO

http://www.flickr.com/photos/aereimilitariorg/3953794205/

http://www.flickr.com/photos/pmtorrone/23362393/


@nxhack

Background:　25年以上情報処理業界　17年間イソターネット業界 Bias:　文系エソジニア　運用な人　*NIXな人

http://twitter.com/nxhack

http://twitter.com/nxhack

伝えたい事: DevOps 運動と何ぞや

対象

WebアプリケーションWebサービスの事業者

スタートアップで成功急成長し、組織作りが必要になった方...

技術系の組織運営にお困りの方...

最近、リリース遅延が常態化しつつある方...

スピード感がなくなってきて、活気が無くなってきたなと思っている方...

衝撃の事実

http://www.etsy.com/

http://www.etsy.com

http://www.etsy.com

http://www.etsy.com/

http://www.etsy.com

http://www.etsy.com

http://twitter.com/allspaw/status/22742505774

◯



11デプロイ/日


◯



15 デプロイ/日

DevOpsのキーワード•WebOps•Agile•Cloud•Realistic-ITIL

Web Operations

http://www.flickr.com/photos/cote/54408562/

http://www.flickr.com/photos/63085612@N00/111068688



TraditionalSystems Operations







http://www.flickr.com/photos/arthur-caranta/2926332140

Web Operations






















Web Operations







Web AppsWeb Services



Shrink WrapXhttp://www.flickr.com/photos/cogdog/116957071/

http://www.flickr.com/photos/cogdog/116957071/

http://www.flickr.com/photos/cogdog/116957071/


The infrastructure is the application.

The application is the infrastructure.







http://omniti.com/seeds/dissecting-todays-internet-traffic-spikes




BURST




WebOps

WebアプリケーションWebサービスの事業者によくある風景…

http://www.flickr.com/photos/johnjoh/368511463/

Startupほんの少しだけ起業論



ScalableGrow fast

ResearchDevelopment

DeploymentRelease

CustomerFeedback

ContinuallyPivot

(Iterate)

Hypotheses Testing

PDCA cycleX

http://www.flickr.com/photos/sdasmarchives/4727993474/

Colonel John (Richard) Boyd (January 23, 1927 – March 9, 1997) was a United States Air Force fighter pilot and Pentagon consultant of the late 20th century, whose theories have been highly influential in the military, sports, and business.



OODA Loop

Continuous DevelopmentContinuous Deployment

ContinuousDelivery

ContinuousDelivery

http://www.amazon.co.jp/dp/0321601912/




ContinuousDelivery

STARTUP CTO’s MISSION

Care about everything...

•Research...•Development...•Operation...(Server, Apps, Storage, Network...)•Facility...(DC, Server, Storage, Network...)•Customer Support...

•everything...

http://www.flickr.com/photos/power_on/5092140218/

COWBOY



http://www.flickr.com/photos/foolswisdom/43145184/

RockStar



A HARD DAY'S NIGHT

http://www.flickr.com/photos/ceonyc/3346786138



順調に成長

規模が大きくなる

仕事が回らない

組織作り

IPO準備

会社の転機


COWBOYX


RockStarX

Transition

What VC’s Don’t Tell You:The Transition ‒ Founders Leave

http://steveblank.com/

http://steveblank.com

http://steveblank.com

http://www.flickr.com/photos/krmnclara/4859716369/

IPO準備事業継続性リスク排除

属人化排除内部統制内部牽制機構



http://www.flickr.com/photos/bpt/321626320/

GOVERNANCE



MANAGEMENT

http://www.flickr.com/photos/deemonita/4043301909/



官僚機構は規模が大きくなれば必要だし合理的。内部統制は官僚機構の統治には必須。

しかし

目的と手段を取り違える

頭の悪い人が多いという罠


残念な人が多いという罠

Organizational Chart

Board CEO

COO CIOCTO CFO

Dev Ops

職務分掌規定

Devの役割Opsの役割

http://www.flickr.com/photos/aaronjacobs/64368770/

Developer



FeaturesEvolution


やったぜ!新機能ビルド成功♪

リリースにギリ間に合った!ふぅねむねむ～



http://www.flickr.com/photos/troyholden/4036820768/

Operator



Services Stability


安全・安定

ダメダメ!もっと早く仕様を渡さんかい!



運用さん

http://www.flickr.com/photos/greg-palmer/3490903215/

Ops focuson failure



hardware failure

http://www.flickr.com/photos/purplemattfish/3760706750/



hardware failure


ぐはっ!故障



hardware failure


Avoid SPOF



hardware failure


Avoid SPF

冗長設備の投資対効果を定量的に示してください。



bursting




bursting


ヤフーに貼られたテレビで紹介された



bursting


Capacity Planning

http://www.amazon.co.jp/dp/0596518579





bursting


Capacity Planning


性能設計の根拠と投資対効果を定量的に示してく

ださい。





software failure

http://www.flickr.com/groups/webopsviz/pool/with/4819476138/




software failure


テスト環境では問題なかったぞ




software failure


本番環境と同じテスト環境本番負荷テスト




software failure


Testing????Testing?テスト環境構築の

投資対効果を定量的に示してください。




operation mistake

http://www.flickr.com/photos/proimos/4199675334/



operation mistake

指示書・手順書うがぁ失敗!

operation mistake

orz

operation mistake

http://www.amazon.co.jp/dp/4122018331http://www.amazon.co.jp/dp/0691004129

指示書・手順書





operation mistake

http://www.amazon.co.jp/dp/4122018331http://www.amazon.co.jp/dp/0691004129

実はこれが一番多い






ContinuousDelivery

CHANGE CAUSES OUTAGES

http://www.flickr.com/photos/adjourned/308053363/



http://www.flickr.com/photos/sis/124916325/



http://www.flickr.com/photos/carve/2856365431/



Dev Ops

Dev Ops

Dev Ops

リリースするよ

無理!詳細仕様渡せ

Dev Ops

障害発生!情報くれー

おーーーーい

しーん...

Dev Ops

リリースじゃぽいっ! むむむむ!!

Dev Ops

Wall of Confusion

Dev Ops

Wall of Confusion

俺達が価値を守っているんだ

俺達が価値を作っているんだ

奴等はスピード感に欠ける・遅すぎ!

奴等はサービスが分かっていない

http://www.flickr.com/photos/rocketjim54/2955889085/

アンタのせいだ!



I'm afraidI can't do that.

I'm sorry, DEVs.

Dev vs. Ops


WebOps

AgileCloud

ちょっと待てよ


残念な人が多いという罠

我々の目的って

Dev & Ops

Goal:To Enable

Our Business

We make Our Servicestogether

To Enable

ContinuousDelivery

じゃぁどうする

Agile

Developer

Manifesto for Agile Software Development

We are uncovering better ways of developingsoftware by doing it and helping others do it.Through this work we have come to value:

Individuals and interactions over processes and toolsWorking software over comprehensive documentation

Customer collaboration over contract negotiationResponding to change over following a plan

That is, while there is value in the items onthe right, we value the items on the left more.

http://agilemanifesto.org/

http://agilemanifesto.org







agile and iterative

ディーン・レフィングウェル (著), 玉川憲 (監修, 翻訳)




DevelopersProduct Owners

Testers

DevelopersProduct Owners

Testersたいへんよくできました

OperatorsSystem AdminsDB Admins

Network EngineersDesigners

OperatorsSystem AdminsDB Admins

Network EngineersDesigners

がんばりましょう

どうすんのよ

Operator

Infrastructure:BootstrappingConfigurationMonitoring

agile operations

agile operations?

Infrastructure is HARD

http://www.flickr.com/photos/johnseb/3425464/



Infrastructure is HARD


!



agile Infrastructure

Cloud

IaaSInfrastructure as a Services

Server: EC2Storage: S3, EBSNetwork: ELB, CloudFrontMonitoring: CloudWatchDataBase: RDS, SimpleDB

Pooled IT Resources

On DemandSelf ServicePay as you goFair pricing

API

% ec2-run-instances ami-b232d0db -n 3 --availability-zone us-east-1a! ! ! !RESERVATION r-385c5950 012301230123 defaultINSTANCE i-5bca5a30 ami-b232d0db pending 0 m1.small 2010-04-07T12:25:47+0000 us-east-1a aki-94c527fd ari-96c527ff monitoring-disabled ebs paravirtualINSTANCE i-59ca5a32 ami-b232d0db pending 1 m1.small 2010-04-07T12:25:47+0000 us-east-1a aki-94c527fd ari-96c527ff monitoring-disabled ebs paravirtualINSTANCE i-5fca5a34 ami-b232d0db pending 2 m1.small 2010-04-07T12:25:47+0000 us-east-1a aki-94c527fd ari-96c527ff monitoring-disabled ebs paravirtual

サーバー 3台用意してね☆

Infrastructure as Code

API! API! API! API! API!Infrastructure as Code! Infrastructure as Code! Infrastructure as Code!

http://www.flickr.com/photos/programmerman/2315566040/



Give me an API, or give me DEATH!

http://www.flickr.com/photos/bitchcakes/3895840169/



Bootstrapping:Infrastructure as a ServicesInfrastructure as Code

Realistic-ITIL

agile Infrastructure

を使いこなす

http://www.flickr.com/photos/robenjoyce/4208638969/

Tools!

http://www.flickr.com/photos/anasilva/4204245004/

http://www.flickr.com/photos/anasilva/4204245004/

DevOpsTool Chainshttp://code.google.com/p/devops-toolchain/

http://code.google.com/p/devops-toolchain/


Release

Generalized architecture

Control

Orchestration

Dispatcher

Provisioning

Deploy

Config management

OS boot/install

Artifact repository

Build

CI Server Issue tracker

SCM Repository

Model

Asset inventory

Host naming

Identity

CMDBMonitoring

Events

Trending Reporting

Trending

Workflows

Resources

Topology

Configuration

Code

Sources

Scheduler

Wednesday, June 23, 2010




Release management

Artifact

Build

SCM

Trackerbugzillamantistrac

svncvsgithg

Repository

yum/yastartifactoryarchivaDFS/openEFSHTTTP

eggsgemsperlmod

tgz/ziprpm/deb/pkg/msijar/war/ear

makemavenant/ivy

sbtrakephing



Dev



Monitoring and control

Measurement Instrumentation

Charting, History, SPC

Alarm Mgmt

Op Console, Control

Runbook Automation

nagioszenossopennms

rrdtoolcacti

ControlTierJobschedulerOpenScheduler



Ops



Provisioning

Bcfg2cfengineChefPuppetSmart Frog

Application Service Orchestration

System Configuration

Cloud or VMImage

LaunchOS

Install

KickstartJumpstartCobblerOpenQRMxCAT

XenlxcopenVZEucalyptusKVM

CapistranoControlTierFabricFuncmCollective

Cloud: OS Install:

Command and Control

Configuration

Bootstrapping

Prov

isio

ning

Act

ivity



Dev

Ops



自動化できうるところは全て自動化する

agile Infrastructure を活用しインフラ構築・構成を含めて

自動化する

Configuration:Configuration Management

agile Infrastructure を活用するよう System Architecture を見直す

System Architecture変えないといけない

従来の考え方から脱却新しい考え方が必要

障害対応




MTBFMean Time Between FailureX

Failure as a Service

Failure is a failurebut

embrace failure

Your sitewill fail!

http://www.flickr.com/photos/gaelx/2912634902/



MTTDMean Time To Detect

MTTRMean Time To Resolve

hardware failure




hardware failure


壊れたら捨てる瞬時に入れ替え



bursting




bursting


必要なだけ起動する不要になれば落とす



software failure





software failure


本番環境と同じテスト環境




operation failure




operation failure


自動化



Monitoring:Measurement

VisualizeShare

http://blog.b3k.us/ooda.html

WebOps OODA LoopDiagram

ActDecideOrientObserve

ExecutionPlanningResourcing

AnalysisVisualizationCorrelation

MetricsMonitoringAlertingAlarming

Benjamin Black, [email protected]



Visible DevVisible Ops

http://www.amazon.co.jp/dp/443407296X/

THE VISIBLE OPS HANDBOOK - 見える運用

Kevin Behr (著), Gene Kim (著), George Spafford (著)

官野厚訳出版社: ブイツーソリューション (2006/1/6)



Web OperationsKeeping the Data On Time

By

John Allspaw, Jesse Robbins

Publisher:O'Reilly MediaReleased:June 2010




Opsの役割


Operator



Services Stability

次のステップへ

http://www.flickr.com/photos/alexthomp/3545005410/

Operator



System AvailabilityEfficiency

Dev ♥ Ops

To EnableOur Businesstogether

Culture

http://www.flickr.com/photos/24443965@N08/3285794855/



PeopleProcessTools


DevOpsCultural and

Professional Movement



Question

http://www.flickr.com/photos/statelibraryqueensland/4903167275/



Appendix:John Allspaw (twitter) (blog)・SlideShare・10+ Deploys Per Day: Dev and Ops Cooperation at Flickr　　(どのプレゼンも素晴らしいですが、このスライドは必ず見てください)

Andrew Clay Shafer (twitter) (blog)・SlideShare・DevOps ‒ You’re Doing IT Wrong « fate = will && choice || circumstance

Adam Jacob (twitter)・SlideShare・Velocity 2010 ‒ Adam Jacob on DevOps・Choose Your Own Adventure OSCON 2010 Edition (PDF)

http://twitter.com/allspaw

http://twitter.com/allspaw

http://www.kitchensoap.com/

http://www.kitchensoap.com/

http://www.slideshare.net/jallspaw

http://www.slideshare.net/jallspaw

http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

http://twitter.com/littleidea

http://twitter.com/littleidea

http://stochasticresonance.wordpress.com/

http://stochasticresonance.wordpress.com/

http://www.slideshare.net/littleidea

http://www.slideshare.net/littleidea

http://stochasticresonance.wordpress.com/2010/03/08/devops-youre-doing-it-wrong/

http://stochasticresonance.wordpress.com/2010/03/08/devops-youre-doing-it-wrong/

http://twitter.com/adamhjk

http://twitter.com/adamhjk

http://www.slideshare.net/adamhjk

http://www.slideshare.net/adamhjk

http://www.youtube.com/watch?v=Fx8OBeNmaWw

http://www.youtube.com/watch?v=Fx8OBeNmaWw

http://dl.dropbox.com/u/1557449/choose_your_own_adventure_oscon_2010_edition.pdf

http://dl.dropbox.com/u/1557449/choose_your_own_adventure_oscon_2010_edition.pdf

Appendix:John M Wills (twitter)・SlideShare・What Devops Means to Me

Damon Edwards (twitter) (blog)・DevOps is not a technology problem. DevOps is a business problem.

PodCast:・DevOps Cafe・IT Management Podcast

Conference / UnConference:・DevOps DAYS・OpsCamp・O’REILLY Velocity

http://twitter.com/botchagalupe

http://twitter.com/botchagalupe

http://www.slideshare.net/botchagalupe

http://www.slideshare.net/botchagalupe

http://opscode.com/blog/2010/07/16/what-devops-means-to-me/

http://opscode.com/blog/2010/07/16/what-devops-means-to-me/

http://twitter.com/damonedwards

http://twitter.com/damonedwards

http://dev2ops.org/

http://dev2ops.org/

http://dev2ops.org/blog/2010/11/7/devops-is-not-a-technology-problem-devops-is-a-business-prob.html

http://dev2ops.org/blog/2010/11/7/devops-is-not-a-technology-problem-devops-is-a-business-prob.html

http://devopscafe.org/

http://devopscafe.org/

http://www.redmonk.com/cote/topic/podcasts/itmanagementguys/

http://www.redmonk.com/cote/topic/podcasts/itmanagementguys/

http://www.devopsdays.org/

http://www.devopsdays.org/

http://www.opscamp.org/

http://www.opscamp.org/

http://velocityconf.com/velocity2010/public/schedule/topic/Velocity+Culture

http://velocityconf.com/velocity2010/public/schedule/topic/Velocity+Culture

Appendix:日本語での情報:・すべての道は DevOps へと : DevOps ‒ mizzy.org ‒ Trac・DevOps: OpenSource Provisioning Toolchain : DevOps ‒ mizzy.org ‒ Trac・DevOpsって何？

・クラウドと「DevOps」を理解する ‒ builder by ZDNet Japan

My Blog:・すでにここにある雲 DevOps

このスライド:・http://www.slideshare.net/nxhack

http://trac.mizzy.org/public/wiki/DevOps

http://trac.mizzy.org/public/wiki/DevOps

http://trac.mizzy.org/public/wiki/OpenSourceProvisioningToolchain

http://trac.mizzy.org/public/wiki/OpenSourceProvisioningToolchain

http://www.slideshare.net/mizzy/devops-4156440

http://www.slideshare.net/mizzy/devops-4156440

http://builder.japan.zdnet.com/news/story/0,3800079086,20411692,00.htm

http://builder.japan.zdnet.com/news/story/0,3800079086,20411692,00.htm

http://www.egrep.jp/blog/category/devops/

http://www.egrep.jp/blog/category/devops/

http://www.slideshare.net/nxhack

http://www.slideshare.net/nxhack

FGD 135Peace IS OUR PROFESSION

Technology

DevOps or: How I Learned to Stop Worrying and Love the Cloud