The Phoenix Project: A Novel About IT, DevOps, and Helping Your

Business Win.

Poll

● How many have read “The Phoenix Project”?● What is the cycle time in your org from code to

production?● How many of you have Ops represented in Dev

teams?– How long has Ops people been embedded in Dev

teams?

Who is here today?

● Why are you here?– Name

– Company

– Role

– Interest in book / event / Limited WIP

– What do you want to get out of tonight?

My Objective

● Get more people to read “The Phoenix Project”– Make a difference to more people

● Provide some thinking tools and references– Speed up your learning and adoption

● Learn from you and your experiences

Why is this important?

High Performing DevOps Teams ● They’re more agile

– 30x more frequent deployments

– 8,000x faster cycle time than their peers

● They’re more reliable– 2x the change success rate

– 12x faster MTTR ● MTTR = Mean time to Restore/Repair/recovery

Source: Puppet Labs 2012 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic

7

Buzz about book first caught my attention

Top 100 Books

● http://www.noop.nl/2013/08/top-100-agile-books-edition-2013.html

●

What does Gene Kim Say about the book?

● http://www.youtube.com/watch?v=FpEd-QKVIrw 5 min

The Phoenix Project - The Authors

● Gene Kim– Gene Kim is a multiple award-winning entrepreneur, the founder

and former CTO of Tripwire and a researcher. He is passionate about IT operations, security and compliance, and how IT organizations successfully transform from “good to great”.

● George Spafford– George is an author and speaker, consulting and conducting

training on strategy, IT management, information security and overall service improvement in the U.S., Canada, Australia, New Zealand and China. Co-author of “The Visible Ops Handbook” and “Visible Ops Security,” George is a certified ITIL Expert, TOCICO Jonah and a Certified Information Systems Auditor (CISA).

● Kevin Behr– Kevin Behr is the founder of the Information Technology

Process Institute (ITPI) and the CTO of Assemblage Pointe. Kevin has twenty years of IT management experience and is a mentor and advisor to Chief Executive Officers and Chief Information Officers.

Book Introduction

● Bill is an IT manager at Parts Unlimited. It’s Tuesday morning and on his drive into the office, Bill gets a call from the CEO.

● The company’s new IT initiative, code named Phoenix Project, is critical to the future of Parts Unlimited, but the project is massively over budget and very late. The CEO wants Bill to report directly to him and fix the mess in ninety days or else Bill’s entire department will be outsourced.

● With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.

● In a fast-paced and entertaining style, three luminaries of the DevOps movement deliver a story that anyone who works in IT will recognize. Readers will not only learn how to improve their own IT organizations, they’ll never view IT the same way again.

Cast at Parts Unlimited

● Parts Unlimited: Business Executives – Steve Masters, CEO, acting CIO

– Dick Landry, CFO

– Sarah Moulton, SVP of Retail Operations

– Maggie Lee, Senior Director of Retail Program Management

– Bill Palmer, VP of IT Operations , former Director of Midrange Technology Operations

– Wes Davis, Director of Distributed Technology Operations

– Brent Geller, Lead Engineer

– Patty McKee, Director of IT Service Support

– John Pesche, Chief Information Security Officer (CISO)

– Chris Allers, VP of Application Development

● Parts Unlimited: Board – Bob Strauss, Lead Director, former Chairman, former CEO

– Erik Reid, Board Candidate

– Nancy Mailer, Chief Audit Executive

Why a novel?● Who has read “The Goal”?

● Storytelling Is The Most Effective Means Of Creating A Shared Understanding

● The pattern is everywhere: The core of the solution-selling, good copy-writing and the best TED talks: Describe the problem, show its significance, describe how the problem is solved and the value it will creates.

● Storytelling becomes very important when the problem we’re trying to point out is not fully recognized, or when the solution being proposed flies in the face of common wisdom.

http://itrevolution.com/learn-more-about-concepts-in-phoenix-project/http://www.youtube.com/watch?v=oP3c1h8v2ZQ

Time LineC

P%

S

t ory

Ti m

e1

154%

02 S

ep2

257%

02 S

ep3

3611

%02

Sep

447

14%

03 S

ep5

6218

%04

Sep

674

22%

05 S

ep7

8425

%05

Sep

894

28%

08 S

ep9

104

31%

09 S

ep10

111

33%

11 S

ep11

119

35%

11 S

ep12

125

37%

12 S

ep13

138

41%

15 S

ep14

146

43%

16 S

ep15

154

46%

17 S

ep16

166

49%

18 S

ep17

177

52%

22 S

ep18

182

54%

23 S

ep19

188

56%

23 S

ep20

203

60%

26 S

ep21

216

64%

26 S

ep22

223

66%

29 S

ep23

233

69%

07 O

c t24

238

70%

11 O

c t25

246

73%

14 O

c t26

255

75%

17 O

c t27

264

78%

21 O

c t28

273

81%

27 O

c t29

282

83%

03 N

ov30

293

87%

03 N

ov31

299

88%

03 N

ov32

307

91%

10 N

ov33

314

93%

11 N

ov34

321

95%

28 N

ov35

328

97%

09 J

an

Problems

How to create a “Problem”?

The ERM framework by the Commission of Sponsoring Organizations of the Treadway Commission (COSO) provides a more disciplined and consistent standard against which to implement and assess a company’s ERM program.

http://www.coso.org/documents/coso_erm_executivesummary.pdfhttp://www.youtube.com/watch?v=b7JldvsY7Ac

Strategy

● Strategy: as embodied by the Phoenix Project, which the entire future of the company depends upon. It requires that IT be a core competency.

Reporting

● Accurate financial reporting: as embodied by the third year repeat audit findings around the IT general controls, but also the loss of accounts payable and inventory management systems which prevents the closing the financial books at the end of quarter. Even the payroll failure resulted in inaccurate payroll numbers, which led to financial reporting errors.

Compliance

● Compliance with laws and regulations (e.g., SOX-404, PCI DSS): failing the SOX-404 audit results in an adverse footnotes by the external auditor in the SEC 10-K statement, and there are grave implications for failing the PCI DSS assessment.

Operations

● Operations (e.g., IT application and project delivery, IT operations, information security)

● The Project Phoenix was $20 million over-budget and three years late, and when it finally was deployed into production, everyone would have been better off it hadn’t. Furthermore, virtually all of the critical business systems that run daily operations require that IT services be running correctly, which they often weren’t.

Generic problems – Downward Spirals

● Fragile artefacts become more fragile● Technical debt grows● Date-driven application projects focus only on features, sacrificing

non-functional requirements, which results in more fragile artifacts in production

● Application deployment take longer, more difficult, getting gets worse● IT Ops is stuck fire fighting and therefore cannot do preventive work

or new projects● Long feature delivery cycle times result in more political decision

making, meaning more focus on features (vs. non-functional requirements)

Hand out time line

Breakthrough 1 - 3

● Breakthrough 1: discovering too much WIP in IT Operations and gob-smacking amounts of reliance on Brent for both project and recovery work

● Breakthrough 2: elevating preventive work to prevent unplanned work (especially for Brent) and making work visible

● Breakthrough 3: throttling the flow of work into IT Operations by the project freeze (subordinate)

Breakthrough 4 - 6

● Breakthrough 4: identifying and limiting the flows of work to Brent (it was genuinely surprising to find out after the fact at how much this resembles the green/red tag pattern used in “The Goal”) Kanban Board

● Breakthrough 5: documenting and defining standardized work, and managing handoffs to reduce time spent in queue (through use of kanbans), and further reducing reliance on Brent

● Breakthrough 6: correctly identifying reliance on IT systems to hit Dick’s corporate objectives (via Gartner RVM model) and correctly scoping audit and infosec (via GAIT and GAIT-R)

Breakthrough 7 & 8

● Breakthrough 7: building DevOps flow of work to better design in non-functional requirements in Dev, reduce batch sizes and enable single-piece flow through Development and IT Operations and better enable operational resilience (replicating the work described by Jez Humble, David Farley, Paul Hammond and John Allspaw)

● Breakthrough 8: when they discover the constraint moves outside the organization, they bring back those resources in-house (replicating one of my favourite Theory of Constraints plays: when there’s idle plant capacity, it’s often cheaper to fabricate parts in-house than to pay a supplier. Lovely.)

4 types of work

● Business project work, ● IT project work, ● changes and ● unplanned work.

Thee Three Ways

● Way No. 1: The flow● Way No. 2: Consistent feedback● Way No. 3: Continual learningPage 99 – Erik Introduces the concept.

Page 297 – Bill learns how important The Three Ways are.

“But I learned that the practices that Allspaw and Hammond espoused are the inevitable outcome of applying the Three Ways to the IT value stream. It totally changed how we managed IT and it saved our company. “How did they do it?” I ask, dumbfounded.”

Way No. 1: The flow

1. Understand the flow of work. ● Without understanding, any changes will have random

effects.

2. Always increase the flow

3. Never pass on defects downstream

4. Never allow local optimization to cause global degradation

5. Achieve profound understanding of the entire system

Way No 2: Consistent Feedback

● Understand and respond to the needs of all customers, both internal and external.

● Shorten and amplify all feedback loops. – Kim said this rule is modelled after the Toyota Lean

manufacturing line, where any employee can stop the line immediately if they see any defect.

● Create quality at the source. – This means pro-actively building quality into everything. There's

no need for developers to rely on QA to find their errors.

● Create and embed knowledge where we need it.

Way No. 3: Continual learning

● Create a culture that encourages experimentation

● Learns from failure● Recognizes repetition as the prerequisite to

mastery

Thanks

● Thanks Barclays Bank for providing the venue● Thanks to the authors of “The Phoenix Project”● Thanks for your participation

References

● http://itrevolution.com/learn-more-about-concepts-in-phoenix-project/

● http://itrevolution.com/a-personal-reinterpretation-of-the-three-ways/

● Free 170 page excerpt:http://itrevolution.com/the-phoenix-project-excerpt/

●

● http://slideshare.net/realgenekim

● TOCICO http://www.tocico.org

● Dr Holt's TOC courses at Washington State University

– http://public.wsu.edu/~engrmgmt/holt/WSU-TOC-Flyer-Web.htm