Upload
travis-donselman
View
51
Download
4
Tags:
Embed Size (px)
Citation preview
Case: CareGroupCase: CareGroup
Steven AdamsTravis Donselman
Ruwani Palihapitiya
9 November 2009
Case: GroupCare/Adams et al. 1
SWOT AnalysisSWOT Analysis
Case: GroupCare/Adams et al. 3
Strengths Weaknesses
• Halamka’s leadership • Y2K Disaster Recovery Plan• Info. Week’s #1 IT organization• 90% cut in capital expenditures• 2nd largest healthcare system in region
• Outdated network components• Weak financial statements• Inadequate security controls• Inadequate change controls• Lack of networking knowledge• Outdated Disaster Recovery Plan
Opportunities Threats
• Partnership with Cisco• Recovering economy (Q4, 2002)• Vendor to Waltham and Glover• Synergies from merger• Further cost cuts due to leveraging IT• Many hospitals closing, leading to less market saturations
• Volatile time for industry• Stagnant industry growth • Mass. Gen.-Brigham and Women’s merger
Symptoms and ProblemsSymptoms and Problems
Case: GroupCare/Adams et al. 4
Symptoms Problems Evaluation Criteria
Consolidation in industry Mass. Gen.-B&W Hosp.Price Pressure HMOs want cost cuts Hospitals losing money MAH, BIDH, NEBH, WH
Excess Supply in health care industry < 60% bed occupancyFierce Competition Many hospitals closing
Efficiency increase of 20% Debt-equity ratio under 1 Cost cuts limited to 5% All CG hospitals earn profit for Q4, 2003 10% ROE for FY 2005
Uncertainty about future New CEO, CFO, CIOLack of centralized IT system Legacy IT systems
New business strategy Merger of different hospitals MAH, BIDH, NEBH, DNH, DGH, WH
95% of personnel know CG mission and departmental goals and objectives All hospitals on Meditech software by end of 2001
Collapse of IT network Out of spec network Layer 2 based Lack of IT knowledge No network specialistLack of Controls No record of changes Napster-like attack
“Five nines” uptime for a period of two years No unscheduled downtime for two years Network is “in-spec” at all times for two years Meet Gartner benchmark
Theory and ConceptsTheory and Concepts
Case: GroupCare/Adams et al. 5
HIGH PATH TO BUSINESS TRANSFORMATION
IT Impact on Core Capabilities
Business Process Design/Reengineering Centralized IT System Improved efficiency
Business Transformation Merger of five hospitals Greater contracting power against HMOs
Incremental Improvement Use of synergies: strong image (Beth Israel), lower debt costs
Emerging Opportunities Vendor to divested Waltham and Nashoba branches Reorganization of Beth Israel and Deaconess-Glover
LOW IT IMPACT ON CORE STRATEGY HIGH
Timeline of Network Timeline of Network CollapseCollapse
Case: GroupCare/Adams et al. 6
Napster-like internal attack
Wed., afternoon
Service restoredThurs., 4:00 am
Intermittent failures
Thursday
CAP declared Thurs., 4:00pm
Network blocked; backup procedures
implemented Fri.,
morning
Redundant core builtSaturday
Network fully
operational Sunday,
night
The network had grown haphazardly due to the Caregroup merger, the standardization of Meditech in all hospitals, and the lack of change controls and networking knowledge. Significantly, legacy applications required a flat topology.
Additionally, the use of aged outdated networks components contributed to the failure and prolonged the recovery process. This is a result of not meeting the Gartner benchmark for IT spending as a percentage of revenue (1.9% vs. 2.7% target).
Reboot of core and Layer 3
Friday
Case: GroupCare/Adams et al. 7
RenaissanceParkswitch-rca switch-rcb
switch-rcc
5500 5500
5500
EastCampus
switch-ccell118
switch-rob05
switch-ly030
SiSi
switch-br203
5500
5509
5500
5500ATM7/1
FEC 9/1-2
FEC10/5-6
WestCampus
switch-spg06b
switch-spg06a
switch-ccw00m4
5500
5500
5500
SiSi
ATM 5/1
SiSi
FEC 8/1-3
FEC 10/1-4FEC 8/3-4
FEC 9/1-4
FEC 8/1-2
FEC 8/1-2
SiSi
FEC 10/1-2
SiSi
FEC 11/5-6
FEC 8/1-2
FEC 9/1-2
FEC 10/5-6FEC10/1-4
FEC 11/1-2
FEC 11/3-4
ATM10/1ATM10/1
FEC 6/23-24 FEC 6/23-24
FEC 3/21-24FEC 4/21-24
FEC 3/21-24FEC 4/21-24
14
8 12
12
12
12
1212
8 12
14
14 14
12
8 8
8
(8) 5505 Switches
(1) 5505 Switch
(15) 5505 Switches(1) 5500 Switch
(37) 5505 Switches
(1) 5500 Switch(21) 5505 Switches(2) 5509 Switches
(1) 3500 XL Switch (PACS)
(18) 5505 Switches
(21) 5505 Switches(1) 3500XL
(4) Ren Ctr (rc5, rc6, rc7, rcc)(1) Mount Auburn (Remote)
(1) 5500 Switch(1) 5505 Switch
(3) 5509 Switches
SiSi
(12) CC East Campus(4) HIM
(3) 109 Brookline Ave(2) 2127 Burlington(3) Research North
SiSi
(24) CC West Campus - - 1 is Dual Homed w/spg06b(1) PACS
(1) Research East
(1) 5500 Switch(1) 5505 Switch(1) 5509 Switch
(4) Dana(3) East(7) Feldberg(2) Finard(3) Kirstein(4) Reisman(5) Rose(1) Service(5) Stoneham(2) Yamens
(1) Baker(4) Deaconess(7) Lowry Medical(1) Maintenance(3) Palmer(1) CC West (Dual-homed w/ccw00m4)
(13) Farr(6) Kennedy(2) Lowry Medical(1) Masco
(3) Ren Ctr (rc7, rc8, rcc)
SiSi
ATM OC-3 (155Mbps) over Sonet
ATM OC-3 (155Mbps) dark fiber
Fast Etherchannel (400 Mbps)Fast Etherchannel (800 Mbps)
Not Active
SiSi
Technical ExplanationTechnical Explanation Layers 2 and 3 were out of specification with 802.1d
standards:
◦ The standard calls for a maximum of 7 hops.
◦ Multiple bridges were 10 hops from the bridge root
◦ Bridge loops present
◦ Insufficient use of routers
Case: GroupCare/Adams et al. 8
802.1d Standards CG Network
One VLAN per subnet One VLAN for multiple subnets
Limited or no bridging Extensive use of bridging
Switching limited to Layer 2 Switching used for Layers 2 and 3
Alternatives and Alternatives and EvaluationEvaluationAlternatives Pros ConsMitigation• Disaster Recovery Plan• Emergency Access System
• Limited access available during outages• Outages run more smoothly
• Moderate capital expenditures• Moderate time requirements• Limited risk involved
Avoidance• Contract with Cisco• Upgrade Components• Sec./Change Controls• SETA
• Minimizes risk of outages• Expertise and knowledge of Cisco• Changes are documented• Employees are less likely to harm the network
• High expenditures $300,000/yr. for Cisco• High level of network knowledge required• Significant time investment• Some uncertainty involved
Acceptance• Do nothing
• No project planning• No capital expenditures• No time required
• System subject to future failures• Threat to company’s reputation and profitability
Transference• Outsource Network Management
• Skilled administrators• Can focus on core competency (knowledge management)
• Need to trust another company• Very expensive• Loss of control over process Case: GroupCare/Adams et al. 9
RecommendationsRecommendationsUse multiple spanning tree protocol Implement monitoring and diagnostic tools to
maintain the health and hygiene of the network:◦ NATkit◦ CiscoWorks◦ HP Network Node Manager
Implement exclusively routers in Layer 3Replace all legacy applicationsReview and, if necessary, replace Meditech PAC
softwareUse dense multicast instead of broadcastHire network VP, network managers, network
administrators, and network technicians
Case: GroupCare/Adams et al. 11
Case: GroupCare/Adams et al. 12
SiSi
Renaissance Park
6500 6500
East Campus West Campus
SiSi
SiSi
SiSi
SiSiSiSi
SiSi
SiSiSiSi
Server Server Server
Data Center
ClosetSwitches
Typical Access LayerConnection to the Core
Typical Access LayerConnection to the Core
Remote LocationWAN Connectivity
Firewall Firewall
Internet
109Brookline
Shapiro ClinicalCtr
Rose
West ClinicalCenter
Palmer
SiSi
SiSiSiSi
Review of CaseReview of Case How did CareGroup respond to the disaster?
◦ Implemented backup procedures◦ Brought in Cisco’s Emergency Response Team◦ Reverted to paper system◦ Installed routers, restored critical systems, reviewed crisis
Your thoughts on how CareGrop handled the crisis.◦ Well organized, decisive action◦ Cisco Emergency Response Team was the right decision◦ Should NOT have changed system before ERT arrived
Your thoughts on network management in light of the case.◦ Vigilant system maintenance and upgrades are critical◦ Control access to the main system at all times◦ Make sure backup plans are in place and well-practiced
Case: GroupCare/Adams et al. 13