Automated Deployment in Support of Continuous Integration to Transform SDLC
Preview:
DESCRIPTION
presentation in IBM Innovate 2013 in Orlando FL
Citation preview
- 1. WebMD Adopts Automated Deployment in Support of Continuous
Integration to Transform their SDLC Teresa Dietrich, Vice President
Technical Operations Derek Chang, Senior Director
- 2. We were here 2
- 3. So we assembled a team 3
- 4. To fix our SDLC 4
- 5. And now everyone is happy. 5
- 6. Technology @ WebMD WebMD, Medscape, MedicineNet, eMedicine,
UK cobrand Serving nearly 1 Billion Pageviews a month, 132 million
unique Running over 200 separate applications, vast majority
in-house developed Environments: Dev/Devint, QA01/02, QA00,
Production/DR Two main data centers, geographically diverse OS: mix
of Linux and Windows Datastores: Sql Server, Oracle, Mongo, Vertica
and mysql Web: mix of Apache and IIS App: mix of Tomcat, ASP, .Net
3.x and 4.x Service: ActiveMQ, Memcache 6
- 7. Before - State of Deployments Build repository a free for
all NAS share; manual upload Package management HP proprietary
package library; manual upload/download Configuration management HP
proprietary CML Deployment module HP software policy Deployment
initiation SBM ticketing system Pre/post deployment tasks custom
scripts and manual steps Dev Sends Email QA Opens Ticket Lead
Assign Ticket Engineer Opens SubTickets DBA Runs Script Engineer
deploys build QA Smoke Tests Ticket Closed 7
- 8. Before Deployment Metrics year total # of work days average
daily max date 2011 2650 251 10.5 34 7/1/2011 2012 3174 251 12.6 50
9/20/2012 Average time to deploy: 55 minutes 8% of tickets rejected
thus required clarification Average build deployment per day: 11.6
Daily Max: 50 8
- 9. Before - Pain Points Require multiple systems Relies heavily
on email for communication Inconsistent environment setup Only Ops
did deployments in QA and Prod Error prone Wrong information
provided in ticketing system Tons of customized scripts Complexity
of deployment procedures Manual steps involved in pre and post
deployment High maintenance cost Configuration management Host
inventory Deployment scripts 9
- 10. Becoming Agile Hired an Agile consulting group Signed up
for Software to manage Agile Dev Trained Scrum Masters Learned to
write user stories, plan sprints and used colored Post-Its
Scheduled daily Stand-ups. Woohoo, now we are officially Agile!
10
- 11. Agile Implications for Deployments 11
- 12. Transform our SDLC 12
- 13. Waterfall vs Agile 13
- 14. Transform our SDLC 14
- 15. Promote Once vs Promote Continuously 15
- 16. Transform our SDLC 16
- 17. Static vs Elastic 17
- 18. Continuous Integration Team Formed a CI Team w/
representatives from: Dev WebOps QA DBA Platform Ops Each team
member had to have day to day hands-on responsibilities. Each team
member had to have the authority to make decisions on behalf of the
organization they represented. Each team member had to be able to
commit to at least 4 hours a week . Team reported back to Sr Mgmt
at specified milestones. 18
- 19. Measuring Success Automated Unit Testing Build Ready to
Build Deployed (by env) Successful Build Deployment (by env)
Successful Automated Smoke test (by env) Build Deployment Execution
time (by env) Build from Dev to Production time 19
- 20. Automated Deployment Sr Management created a document with
1 page on assumptions and 1 page on milestones to kick off the team
on their first assignment. Assumptions included: Standards
developed for: build, packaging, config, naming and monitoring
Roles and Responsibilities by Environment Deployment success and
Environment Exit criteria 20
- 21. Evaluation Process # Procedure Working group Management P1
principles and assumptions P2 define evaluation criteria X P3
define baseline functionalities and features collect list of
products to be evaluated finalize product evaluation form and list
to be evaluated X research products based on criteria review/rank
products X X X P9 P10 review and finalize list of products for POC
vendor demo In house POC X X X P11 review POC results X Description
X P4 P5 P6 P7 P8 X X X referrals, forums X 5-7 products vendor
documentation, forums and user community X POC should include
must-have functionalities defined in F11 below 2-3 vendors 1
product at a time 21
- 22. Most Important Features Efficient configuration management
Host inventory Capability of easy code promotion, backup and
rollback Comparison artifact/environment/configuration
Orchestration engine reusability and visibility Capability to
integrate with other systems Support 22
- 23. Developing the Requirements Guidelines from Sr Management
Review and understand the tools we were using. In-depth analysis of
current deployment process Understand our current pain points.
Learn from what products are available on the market. Learn from
what other people are using and doing right. 23
- 24. Functional Requirements # Functionality Category importance
F1 F2 start application stop application operation operation
must-have must-have F3 application configuration management -
administration operation must-have F4 F5 F6 F7 F8 F9 F10
application configuration management - versioning application
configuration management - comparison application build deployment
application build rollback fail-safe protection customizable
post-deployment operations single interface for deployment-related
operations administration administration operation operation
usability operation usability must-have must-have must-have
must-have must-have must-have must-have F11 log/report for
troubleshooting and auditing administration must-have F12 F13 F14
F15 F16 integration administration administration Operation
integration must-have must-have must-have must-have must-have
integration with build server Visibility into deployment operation
Event and hand-off Notification Scheduled Deployment Integration
with configuration management solution 24
- 25. Functional Requirements # Functionality Category importance
F17 integration with ticketing system F18 integration with revision
control system integration integration nice-to-have nice-to-have
F19 integration with software builder integration must-have F20 F21
F22 F23 F24 F25 F26 integration integration integration operation
administration administration usability nice-to-have nice-to-have
nice-to-have nice-to-have must-have nice-to-have must-have F27
deploy unbundled artifacts from multiple sources usability
must-have F28 F29 F30 F31 usability integration administration
usability nice-to-have must-have must-have must-have Integration
with testing automation solution Integration with defect tracking
solution Integration with availability monitoring database build
deployment security and access control mechanism agentless
architecture native support for scripting language mobile app
deployment clustering database connectivity awareness resource
inventory Support both windows/Linux platform 25
- 26. Evaluation Score Card Danny 3 Carrie 4 Matt 4 SRE 3 Ab Raj
Yong total average Vendor support Vito 3 3 3 5 28 3.5 User
community 5 4 4 5 4 4 3 5 34 4.25 Documentation 4 5 5 5 5 4 4 5 37
4.625 Cost for license* 1 2 2 1 1 1 2 2 12 1.5? Cost for
infrastructure 2 2 3 3 2 2 1 2 17 2.125 Effort for maintenance 3 4
5 5 3 3 1 1 25 3.125 Cost for development and implementation 3 4 3
3 4 3 4 3 27 3.375 Interfaces such as API or web services 5 5 5 3
37 **4.625 Functionality Refer to table 1 5 4 5 4 5 5 5 5 38 4.75
Usability Ease of use 4 4 4 4 4 4 5 4 33 4.125 Learnability 4 4 3 3
4 3 5 4 30 3.75 Product support Cost (lower) Integrability 5 5 5 4
* including derived maintenance charge and yearly support ** scale
will be adjusted prior to vendor evaluation phase 26
- 27. Necessary Standards and Adjustments Change of roles and
responsibilities All configurations go into uDeploy and standards
across apps. Standardize environment setup and rebuild if
necessary. Standardize naming Application/component/environment
naming Versioning and release number Configuration key/value pairs
Agent/resource naming Automatic smoke testing 27
- 28. Identifying All Possibilities Source: referrals,
communities, and conferences Up to 5 were proposed by each member
28 were identified as potential candidates 3 were given to each
team member for initial screening 18 were eliminated for not
meeting critical requirements 10 possible candidates moved to the
second round 28
- 29. Choosing the Contenders Evaluation criteria: function
checklist and vendor documents Priority ranked 10 remaining
products Top 5 for on-site demo and discussion Each vendor was
given criteria and use cases Use scorecard to rate and major
pros/cons comparison Top 2 to participate on-site POC 29
- 30. Proof of Concept POC designed by WebMD Selected 2 most
representative applications Success criteria use cases and
functional requirements Timeline 5 days on-site and 2 weeks
afterward Participated in installation, design and implementation
Daily working session 30
- 31. Final Selection Use case verification and documentation
In-depth review and discussion Pros and cons comparison The team
voted unanimously and chose uDeploy! Evaluation result was
communicated to Sr. Mgmt for final review On-site training for the
entire Technology organization Demonstration along with highlights
of the selection process 31
- 32. Demo 32
- 33. Key Benefits of uDeploy Usability, everything accessible in
one tool Integration capabilities to existing SDLC tools
Application Configuration Management Ease of promoting builds
through environments Orchestration of 3rd party tools and reuse of
templates Visibility process, troubleshooting, reporting, approval
and notification 33
- 34. After Deployment Metrics Duration March/4/2013-June/3/2013
75 components developed around 30% implemented 419 agents in use
1070 build deployments 65 work days 15.1 per day Daily max: 43 on
May/29/2013 Average - End to end time 3 minutes 11 seconds 34
- 35. Lessons Learned Change is hard - redefining roles and
responsibilities Bottleneck just shifted forward, on to the next
challenge. DevOps culture - collaboration between Dev, Ops and QA.
Ease of use, less technological expertise required When a piece of
software is the centerpiece of your SDLC, its worth the work to
find the right solution. Cross functional teams work well when they
have clear direction and authority from management. Integration
with other tools is key. Defining the process, policy and standards
are as important as selecting the right tool. 35
- 36. And now how here we are. 36
- 37. Appendix 1 roles and responsibilities # SDLC environment
Devint QA01/QA02 QA00 Production t1 t2 Environment Prep initiation
(ticket or email to identify need) capacity evaluation/deployment
target designation for new application Dev Dev/Ops QA
Dev/Ops/QA-PERF QA Dev/Ops/QA-PERF Ops Dev/Ops/QA-PERF t3-1 t3-2
Dev Dev ProdOps ProdOps ProdOps ProdOps ProdOps ProdOps t3-3 t3-4
t3-5 t4-1 t4-2 t4-3 t4-4 t4-5 Server provisioning request database
request mssql storage request DNS/VServer provisioning request
middleware installation setup new uDeploy component template for
applications update existing uDeploy component template for
applications application related config variables/definitions app
related config values***** middleware related config
variables/values***** Dev Dev HPSA - ProdOps and dev self service
Dev; signed off by Ops/QA Dev; signed off by Ops/QA Dev; signed off
by Ops/QA Dev Dev ProdOps ProdOps ProdOps n/a n/a n/a QA ProdOps
ProdOps ProdOps ProdOps n/a n/a n/a ProdOps ProdOps ProdOps ProdOps
ProdOps n/a n/a n/a ProdOps ProdOps t4-6 t5 t6 t6-1 t6-2 t6-3 t7
snapshot approval for deployment people who push Deployment button
Troubleshoot if deployment fails Automatic smoke testing
Troubleshoot if smoke testing fails Testing and Sign-off snapshot
implementation will be based on discussion with professional
service Dev Dev Dev uDeploy/Selenium n/a Dev QA QA QA
uDeploy/Selenium QA QA QA ProdOps ProdOps**** uDeploy/Selenium
ProdOps**** QA Ops management ProdOps ProdOps**** uDeploy/Selenium
ProdOps**** QA *template includes component and process templates,
config key/value pairs **e.g. functional testing for QA01/02; UAT
for QA00 ***based on promotion approach; will revisit during
on-site training as well as professional service engagement
****party initiates communication and troubleshooting, which will
involve Dev/Ops/QA ***** values will be contributed by all parties
37
- 38. Appendix 2 provisioning process # 1 Setup uDeploy access**
and perform uDeploy upgrade Step initiator CI 2 Setup devint based
on Ops standard Dev 2.1 Provision new VMs - devint Deploy
infrastructure software and servers 2.2
(IIS/Tomcat/Memcached/ActiveMQ) -devint 2.3 Provision DNS and
Netscaler rules - devint owner PlatformOps ticket firstaid ticket#0
platformOps uDeploy ticket Dev SiteOps firstaid ticket#1 sysops
host provisioning Dev ProdOps firstaid ticket#2 ProdOps general Dev
ProdOps firstaid ticket#3 ProdOps general 2.4 Provision Storage -
devint Dev StorageOps firstaid ticket#4 sysops storage provisioning
2.5 Provision DB - devint Dev DBOps firstaid ticket#5 DBOps general
Dev Dev/ProdOps 3.1 naming standardization Dev/ProdOps Dev/ProdOps
3.2 Component mapping Dev/ProdOps Dev/ProdOps 3.3 Server inventory
Dev ProdOps 3.4 Review/confirm server inventory Dev/ProdOps
Dev/ProdOps 3.5 Install uDeploy agents Dev/ProdOps Sysadmin 3.6
Component/environment mapping Dev/ProdOps Dev/ProdOps 3
Application/component/environment population Dev/ProdOps
Dev/ProdOps 4.1 Collect/Review/consolidate application config 4
Application config Dev/ProdOps Dev/ProdOps 4.2 Designate Config
namespace Dev/ProdOps Dev/ProdOps 4.3 Import config Dev/ProdOps
Dev/ProdOps 4.4 Build server configuration Dev Dev Dev Dev/ProdOps
Dev Dev/ProdOps Dev Dev/ProdOps/DBOps Dev/QA/DBOps/ProdOps
Dev/QA/DBOps/ProdOps QA firstaid ticket#6 sysops general
Dev/ProdOps 5 Process Development 5.1 Develop Application Process
if necessary Develop or Reuse component template
(IIS/Tomcat/Windows 5.2 service/DB deployment,,etc) Designate roles
for each environment (configuration, deployment, 6 approval)*** 7
Set up additional notification if necessary; with product email
alias ** provision user accounts and assign members to each
privileged groups as shown in table 2 below *** admin for each
environment will be responsible to designate roles 38
- 39. Appendix 3 - guidelines Creation of automation templates
should be done at the rollout of the tool and not for each product
or application. Types of templates may include Applications,
Configurations and DB changes. Before a build is automated for the
first time through the tool in DevInt, an existing template must be
identified or a new template must be developed. If a new template
is needed, there should an agreement that none of the existing
templates are appropriate and involvement from a cross functional
team to develop a new one. The Automated Deployment Process begins
in the DevInt environment. The exit criteria from each environment
should be at least one successful automated deployment using the
prescribed tool. Success is defined as a successful unit test in
DevInt or a successful smoke test in QA. Until the tool is
identified, assumption is that the configuration lives with the
build. When QA is referenced below, the assumption is that it is a
member of the new Deployment Management team not a general QA
analyst. We need to define, review and agree to a set of standards
with regards to builds, build packaging, configurations, naming and
monitoring to ensure the success of the automation deployment
process. The groups listed below each environment have the
responsibility of creating and executing the deployment and first
level troubleshooting within that environment. 39
- 40. Appendix 4 demo summary Application:
https://deploy.webmd.net/#application/73d7afd9-ca16-41d8-b67c-5ec00857d0c2
Quick summary of state of component/environment and configuration
Component:
https://deploy.webmd.net/#component/ac0dbc11-ce86-4b87-9623-661575815d2f
History view of who did what where. Environment:
https://deploy.webmd.net/#environment/ac5bcdf6-c412-4ab4-82b9-94c5df514ee4
Where you see version history and request details Host inventory
https://deploy.webmd.net/#environment/ac5bcdf6-c412-4ab4-82b9-94c5df514ee4/inventory
Application property
https://deploy.webmd.net/#application/d7de8f89-d861-458d-81c5-1778f7360dcb/properties
Component property
https://deploy.webmd.net/#component/ac0dbc11-ce86-4b87-9623-661575815d2f/properties
Environment and environment-component property
https://deploy.webmd.net/#environment/ac5bcdf6-c412-4ab4-82b994c5df514ee4/properties
Global/system property: https://deploy.webmd.net/#system/properties
More to come process, resource, version - properties/configuration
management Application process
https://deploy.webmd.net/#applicationProcess/222e0a13-874d-4df1-b5ac-87e69af6f360/13
Manage component operations during deployment sequence, lock,
decision point and approval Component process
https://deploy.webmd.net/#componentProcess/9b7208f5-3190-486c-994e-0922075f7d28/43
Where deployment actions take place stop iis, check application
pool, download artifact, search and replacement, start iis
Component process template Environment specific process
https://deploy.webmd.net/#environment/08dd45de-a528-46ee-b07e5bbf527616c4/approvals
Plugins Rest API Everything is versioned! Snapshot
https://deploy.webmd.net/#application/d7de8f89-d861-458d-81c5-1778f7360dcb/snapshots
40
- 41. Appendix 5-1: application view 41
- 42. Appendix 5-2: process template 42
- 43. Appendix 5-3: component view 43
- 44. Appendix 5-4: component process 44
- 45. Appendix 5-5: Component property 45
- 46. Appendix 5-6: component-environment view 46
- 47. Appendix 5-7: environment property 47
- 48. Appendix 5-8: environment inventory 48
- 49. Appendix 5-9: artifact view 49