Embed Size (px)
Citation preview
1
A Single Pane of Glass Will Never Exist
Session 232, February 23, 2017
Ryan J. Klein, IT - St. Joseph Health
Kelly Nunez, IT - St. Joseph Health
2
Speaker Introductions
Ryan J. KleinSt. Joseph HealthAnaheim, CA714-704-6824www.stjoe.org
3
Speaker Introductions
Kelly NunezSt. Joseph HealthAnaheim, CA714-704-7454www.stjoe.org
4
Conflict of Interest
Ryan J. Klein
Kelly Nunez
Has no real or apparent conflicts of interest to report.
5
Agenda
• Overview of St. Joseph Health
• Evolving infrastructure and IT operations and St. Joseph Health
• Enterprise Monitoring Team
• The Monitoring Opportunity at SJH
• Building our Monitoring Strategy
– People, Process, Technology
• Our Results & Recommendations
6
Learning Objectives
1. Assess the theory of the "single-pane-of-glass" monitoring strategy for
infrastructure and applications
2. Analyze the process improvement steps taken by the SJH Enterprise
Monitoring team
3. Propose a cohesive idea for a infrastructure and application monitoring solution
4. Create a Monitoring as a Service (MaaS) that is repeatable and expandable
7
Benefits Realized for the Value of Health IT
Savings and Satisfaction
• Automated deployment of new systems
• Integrated alert and notification systems
• Reduce cost by 27.3%
• Reduced alert notification time by 73.8%
• Improved incident engagement and
decreased resolution times resulting in
higher application uptime and decrease
clinical impact
8
Overview of St. Joseph Health
9
Providence St. Joseph Health at a glance
10
Strategic Principles
St Joseph Health IT Infrastructure & Operations StrategyVision:
Information Technology drives competitive
advantage, enhances the healthcare experience,
and provides an operationally streamlined
service.
Mission:
We provide solutions and an environment that:
• Meet and exceed the needs of our health
system
• Are agile and high-performing
• Promote collaborative innovation
• Provide proactive and measurable value
11
Evolving Infrastructure and IT Operations and St. Joseph Health
12
Two Types of IT
Managing two separate, coherent modes of IT delivery, one focused on stability and the other on agility.
Traditional IT DevOps
• Safety• Accuracy• Silo Teams• Waterfall Methodologies• High Touch• Traditional Governance
• Safety• Speed• Cross Functional Teams• Agile Methodologies• Low Touch (Automation)• New Governance
13
A Service Line approach
14
SJH Service Inventory Overview
15
Enterprise Monitoring TeamVision:
Immediately identify and alert on issues to be quickly resolved before patient care is impacted and further researched on how to be prevented from reoccurring.
Mission:
Provide a strong monitoring solution that proactively engages technical teams to predict and prevent issues.
16
Monitoring Strategy Review
Previous Method
• Team focused/no engagement
• Reactive approach
• Tool overlap with capability gap
• Lack of visibility to new deployments
• No end to end (E2E) monitoring
• Lack of vendor engagement
• No governance
• Limited capabilities for Cloud and SaaS
• Not conducive to DevOps requirements
New Method
• Application focused/fully engaged
• Proactive approach
• Consolidate tools and fill gap
• Implement during provision
• E2E monitoring
• Engage vendors
• Implement governance
• Incorporate Cloud and SaaS architectures
• Enable support for DevOps
Strategy Focus:
Deployment Approach w/Component Monitoring
Strategy Focus:
Applications Approach w/E2E Infrastructure Monitoring
17
Key Areas of Monitoring Focus
End to
End V
iew
Event C
orre
latio
n
Hybrid Public SaaS
18
Enterprise Monitoring Processes
Technology
Process
People
Monitoring, IT, Project Governance and Security
Proactive Response & Reporting
Current and new tools
19
Change Monitoring Team FocusTool focused -> Application focused
• Increase application accountability within the team
• Build monitoring knowledge experts by application
• Tier 1 applications
– Primary and Secondary Champions
• Tier 2 and 3 applications
– Primary Champion
– Knowledge transfer throughout team People
20
Implement Monitoring Checkpoints
Governance Teams Integration
IT Project Governance Team:
• Technical Assessment
IT Architects Team:
• IT Architecture Review
Security Team:
• Security Assessment
Support Teams Awareness
• IT Support Teams
• Application Teams
• Incident Response Teams
• Service Desk
• NOC
People
21
Overcoming Challenges• Application teams lacked knowledge of application architecture
– Worked directly with vendor
– Engaged teams often and frequently
• Teams functioned in silos
– Became liaison between application and technical teams
– Marketing campaign to explain MaaS program
People
22
Implement E2E Monitoring ModelImplement Proactive Response
• Integration with ticketing system (standard)
• APM and synthetic monitoring with immediate notifications to application teams
• Auto-baselining infrastructure performance with immediate notification to application/technical teams
• Additional notification levels options (email, text)
Strengthen Reactive Response
• Integration with Mission Control and Incident Response
• Mandatory monitoring team participation on all calls
• IT Liaison for Service Desk and NOC
• Transparency of all alerts and dashboards
Process
23
Implement Monitoring Checkpoints
Prevent going around the process
Governance Processes:
• Technical Assessment
• IT Architecture Review
• Security Assessment
IT Processes:
• Installation/decommission infrastructure
• New site baseline testing
• Authorized requesters
• Proactive response
Increase transparency
Application Performance:
• Alerts, baseline health
Infrastructure Performance:
• Alerts, service/process health, device state,
baseline health, monitor state
Published Dashboards/Reports:
• Mission Control (Incident Response)
• Executive summary
• Operational details
Process
24
Overcoming Challenges• Support Teams didn’t want to change their process
– Marketing campaign to explain MaaS program
– Worked tightly with teams to update processes
• IT Liaison for Service Desk and NOC didn’t exist
– Created process
– Provided training and access
Process
25
Tool Comparison• Identify components
• Identify availability/performance monitoring
• Map to current tools
• Identify the gap
– Missing because not implemented yet?
– Missing because not purchased yet?
Technology
26
Identified Gaps• DevOps integration
– Integrate with vRealize Automation and Puppet – Automate deployment of agents (servers)– NextGen Network project engagement (automate deployment network)
• Limited ESX host monitoring– vRealize Operations engagement– Reclaim funds by replacing current ESX host monitoring solution
• Limited storage monitoring– Support teams and vendor engagement– Tool review and implementation
• Limited knowledge and training– vRealize Operations training– Tableau training
Technology
27
Overcoming Challenges• Limited budget for training
– Sent half team for training
– Knowledge transfer and documentation
• Lack of budget for new tools
– Contract negotiations current tools
– How close to 100% could we get with current tools? 80/20
– Management accepts remaining risk
– Reclaim funds from overlapping tools Technology
28
Results: Monitoring as a Service (MaaS) Program
• Created a repeatable and standardized service for our Monitoring Framework.
• Incorporates automatic deployments, notifications, SJH standards, vendor
recommendations and industry best practices.
• Documents application topology and interdependencies of each application.
• Finds “best fit” monitoring for each tier application.
• Provides full service reporting capabilities to all users with total transparency.
• Aligns with Incident Management and integrates with DevOps and CMDB.
• Ensure all aspects of an application are monitored and alerting.
29
Watch Analyze Integrate
Verification of Application Workflow
All Upstream and Downstream Applications Documented
Infrastructure List verification
Complete topology including storage and physical servers
Alert notification and Ticketing
Default Monitoring thresholds
All Service, Processes, Logs, File or Folders Monitored
Vendor Recommendations
Custom Monitoring threshold Normalization
Synthetic Script
Other Monitoring Tools
Mandatory for all Mandatory for Tier 1 Available for all
MaaS Options Menu
30
31
Benefits Realized for the Value of Health IT
Savings and Satisfaction
• Automated deployment of new systems
• Integrated alert and notification systems
• Reduce cost by 27.3%
• Reduced alert notification time by 73.8%
• Improved incident engagement and
decreased resolution times resulting in
higher application uptime and decrease
clinical impact
32
Recommendations• Begin with the consumer and not the technology.
• Become user and application centric.
• Obtain a tool set that works for you. Unlike one ring, there is no one-tool to rule them all.
• Make sure tools are flexible, sustainable and supports both the traditional and DevOps environments.
• Be proactive.
• Collaborate with hardware and software support teams and vendors.
33
Questions
*Please complete the online session evaluation
Ryan J. Klein
St. Joseph Health
Kelly Nunez
St. Joseph Health
