Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops

  • View
    1.106

  • Download
    0

Embed Size (px)

Text of Kim IT Pro Forum Eugene: IT at Ludicrous Speeds - rugged dev ops

  • 1.IT At Ludicrous Speeds:Rugged DevOps and MoreGene KimAuthor, Visible Ops HandbookIT Pro Forum EugeneJuly 17, 2012Session ID: @RealGeneKim, genek@realgenekim.me

2. Where Did The High Performers Come From?@RealGeneKim, genek@realgenekim.me 3. Visible Ops: Playbook of High Performers The IT Process Institute hasbeen studying high-performingorganizations since 1999 What is common to all the highperformers? What is different between themand average and lowperformers? How did they become great? Answers have been codified inthe Visible Ops Methodology www.ITPI.org@RealGeneKim, genek@realgenekim.me 4. DevOps:Engage Ludicrous Speed! @RealGeneKim, genek@realgenekim.me 5. Source: John Allspaw @RealGeneKim, genek@realgenekim.me 6. Source: John Allspaw @RealGeneKim, genek@realgenekim.me 7. @RealGeneKim, genek@realgenekim.me 8. Source: John Allspaw @RealGeneKim, genek@realgenekim.me 9. Source: John Allspaw @RealGeneKim, genek@realgenekim.me 10. Source: Theo Schlossnagle@RealGeneKim, genek@realgenekim.me 11. Source: Theo Schlossnagle@RealGeneKim, genek@realgenekim.me 12. Source: Theo Schlossnagle@RealGeneKim, genek@realgenekim.me 13. Source: John Jenkins, Amazon.com @RealGeneKim, genek@realgenekim.me 14. Ludicrous Speed! 16@RealGeneKim, genek@realgenekim.me 15. Ludicrous Fail?! 17@RealGeneKim, genek@realgenekim.me 16. @RealGeneKim, genek@realgenekim.me 17. Source: James Wickett@RealGeneKim, genek@realgenekim.me 18. Why DevOps Is So Important To Me@RealGeneKim, genek@realgenekim.me 19. Since 1999, Weve Benchmarked 1500+IT Organizations Source: EMA (2009) Source: IT Process Institute (2008)@RealGeneKim, genek@realgenekim.me 20. High Performing IT Organizations High performers maintain a posture of compliance Fewest number of repeat audit findings One-third amount of audit preparation effort High performers find and fix security breaches faster 5 times more likely to detect breaches by automated control 5 times less likely to have breaches result in a loss event When high performers implement changes 14 times more changes One-half the change failure rate One-quarter the first fix failure rate 10x faster MTTR for Sev 1 outages When high performers manage IT resources One-third the amount of unplanned work 8 times more projects and IT services 6 times more applicationsSource: IT Process Institute, 2008 @RealGeneKim, genek@realgenekim.me 21. 2007: Three Controls Predict 60% OfPerformance To what extent does an organizationdefine, monitor and enforce the following? Standardized configuration strategy Process discipline Controlled access to production systems@RealGeneKim, genek@realgenekim.me Source: IT Process Institute, 2008 22. Tough Love From Ari Balogh @RealGeneKim, genek@realgenekim.me 23. The Downward Spiral Operations Sees Dev Sees Too many fragile and insecure More urgent, date-driven projects applications in production put into the queue Too much time required to restore Even more fragile code (less servicesecure) put into production Too much firefighting and unplanned More releases have increasingly work turbulent installs Planned project work cannot complete Release cycles lengthen toamortize cost of deployments Frustrated customers leave Bigger deployment failures Market share goes down More time spent on firefighting Business misses Wall Street commitments Ever increasing backlog of workthat cold help the business win Business makes even larger promises to Wall Street Ever increasing amount oftension between ITOps, Development, DesignThese arent ITSM or IT Operations problemsThese are business problems!@RealGeneKim, genek@realgenekim.me 24. My Mission: Figure Out How Break The IT Core Chronic Conflict Every IT organization is pressured to simultaneously: Respond more quickly to urgent business needs Provide stable, secure and predictable IT service Words often used to describe process improvement:hysterical, irrelevant, bureaucratic, bottleneck, difficult to understand, not aligned with the business, immature, shrill, perpetually focused on irrelevant technical minutiaeSource: The authors acknowledge Dr. Eliyahu Goldratt, creator of the Theory of Constraints andauthor of The Goal, has written extensively on the theory and practice of identifying and resolvingcore, chronic conflicts.26@RealGeneKim, genek@realgenekim.me 25. Good News: It Can Be DoneBad News: You Cant Do It Alone @RealGeneKim, genek@realgenekim.me 26. Ops@RealGeneKim, genek@realgenekim.me 27. QA And Test Source: Flickr: vandyll @RealGeneKim, genek@realgenekim.me 28. Development@RealGeneKim, genek@realgenekim.me 29. Process And Controls @RealGeneKim, genek@realgenekim.me 30. Product Management And Design Source: Flickr: birdsandanchors @RealGeneKim, genek@realgenekim.me 31. DevOps: Its A Real Movement I would never do another startup that didntemploy DevOps like principles Its not just startups its happening in theenterprise and in public sector, too I believe working in DevOps environments willbe a necessary skillset 5 years from now Agile helped Dev regain trust with the business;DevOps will help all of IT IT becoming more automated relies on DevOpspractices (especially PaaS)@RealGeneKim, genek@realgenekim.me 32. If I Could Wave A Magic Wand, Everyone Will Become conversant with DevOps and recognizethe practices when you see them Be energized about how ITSM practitioners cancontribute in this organizational journey Leave with some concrete steps to get somegreat outcomes Become a part of a team that starts puttingDevOps practices into place 34@RealGeneKim, genek@realgenekim.me 33. How Do You DoDevOps?35 @RealGeneKim, genek@realgenekim.me 34. The Prescriptive DevOps Cookbook DevOps Cookbook Authors Patrick DeBois, Mike Orzen,John Willis Goals Codify how to start and finishDevOps transformations How does Development, ITOperations and Infosecbecome dependable partners Describe in detail how toreplicate the transformationsdescribe in When IT Fails: TheNovel @RealGeneKim, genek@realgenekim.me 35. The Goal by Dr. Eliyahu Goldratt@RealGeneKim, genek@realgenekim.me 36. 38 @RealGeneKim, genek@realgenekim.me 37. 39 @RealGeneKim, genek@realgenekim.me 38. The First Way:Systems Thinking @RealGeneKim, genek@realgenekim.me 39. The First Way:Systems Thinking(Business) (Customer) @RealGeneKim, genek@realgenekim.me 40. The First Way:Systems Thinking (Left To Right) Dont pass defects downstream Dont optimize locally Always increase flow: elevatebottlenecks, reduce WIP, throttle release ofwork, reduce batch sizes Understanding where reliance is placed@RealGeneKim, genek@realgenekim.me 41. Phase 1: Extend the Agile CI/CR Processes Create one-step Dev, Test and Productionenvironment creation procedure in Sprint 0 Create the one-step automated codedeployment procedure Properly integrate release, configuration andchange into the value stream (as well as QA andinfosec) Ensure developers dont leave until productionchange is successful Assign Ops person into Dev team@RealGeneKim, genek@realgenekim.me 42. Definition: Kanban Board Signaling tool to reduce WIP and increase flow44 @RealGeneKim, genek@realgenekim.me 43. The First Way:Systems Thinking: ITSM Insurgency Have someone attend the daily Agile standups Gain awareness of what the team is working on Find the automated infrastructure project team (e.g.,puppet, chef) Release managers can provide hardening guidance Integrate and extend their production configuration monitoring Find where code packaging is performed Integrate security testing pre- and post-deployment Integrate testing into continuous integration and releaseprocess Add security test scripts to automated test library Define what changes/deploys cannot be made withouttriggering full retest@RealGeneKim, genek@realgenekim.me 44. The First Way:Outcomes Determinism in the release process Creating single repository for code and environments Consistent Dev, QA, Int, and Staging environments, allproperly built before deployment begins Decreased cycle time Reduce deployment times from 6 hours to 45 minutes Refactor deployment process that had 1300+ steps spanning 4 weeks Faster release cadence@RealGeneKim, genek@realgenekim.me 45. The Second Way:Amplify Feedback Loops @RealGeneKim, genek@realgenekim.me 46. The Second Way:Amplify Feedback Loops (Right to Left) Expose visual data so everyone can see howtheir decisions affect the entire system Get Development closer to Operations andcustomers Create a reliable system system of work thatimproves itself @RealGeneKim, genek@realgenekim.me 47. Phase 2: Extend Release Process And CreateRight -> Left Feedback Loops Embed Dev into Ops escalation process Invite Dev to post-mortems/root cause analysismeeting Have Dev cross-train IT Operations Ensure application monitoring/metrics to aid inOps and Infosec work (e.g., incident/problemmanagement@RealGeneKim, genek@realgenekim.me 48. The Second Way:Amplify Feedback Loops: ITSM Insurgency Find areas in the incident and problemmanagement processes where Developmentknowledge could help Ensure that countermeasures are captured inthe Agile backlog Find that developer who really cares about theproduction environment @RealGeneKim, genek@realgenekim.me 49. The Second Way:Outcomes Defects and security issues getting fixed fasterthan ever Reusable Ops and Infosec user stories now partof the Agile process All groups communicating and coordinatingbetter Everybody is getting more work done@RealGeneKim, genek@realgenekim.me 50. The Third Way:Culture Of Continual Experimentation AndLearning @RealGeneKim, genek@realgenekim.me 51. The Third Way:Culture Of Continual Experimentation AndLearning Foster a culture that rewards: Experimentation (taking risks) and learning fromfailure Repetition is the prerequisite to mastery Why? You need a culture that keeps pushing into the dangerzone And have the habits that enable you to survive in thedanger zone @RealGeneKim, genek@realgenekim.me 52. Help IT Operations The best way to avoid failure is to fail constantly Hard