Creating an Actionable Disaster Recovery Plan
2
Disaster Planning StrategyDisaster Planning Strategy
Presentation OutlinePlan Justification
Disaster Definitions & FactsCosts of a DisasterBenefits of Planning
Building an Actionable Disaster Recovery PlanProgram InitiationRisk AssessmentDetailed Risk AssessmentDisaster Recovery PlanMaintenance PlanTest Plan & Results
3
Disaster Planning StrategyDisaster Planning Strategy
Plan Justification
4
Disaster Planning StrategyDisaster Planning Strategy
What’s a disaster?
“A disaster is an occurrence that disrupts the functioning of an organization resulting in the loss of data, loss of personnel, loss of business or loss of time” – Hiatt 2000
5
Disaster Planning StrategyDisaster Planning Strategy
Disaster FactsCommon IT disasters:
Power outages 28%Storm damage 12%Floods 10%Hardware error 8%Bombing 7%Hurricanes 6%Fires 6%Software errors 5%Power surge/spike 5%Earthquake 5%
6
Disaster Planning StrategyDisaster Planning Strategy
TermsBusiness Continuity Planning
Advance planning and preparations to ensure continuity of critical business functions
Disaster RecoveryAdvance planning and preparations to minimize loss and facilitate recovery of core IT assets
7
Disaster Planning StrategyDisaster Planning Strategy
Tangible and Intangible CostsPatient care and patient safetyPaying staff who are idleAdded work, related to manual operationsOther hard cash costsLost businessLost customer loyalty – your reputation!
8
Disaster Planning StrategyDisaster Planning Strategy
Recovery Planning BenefitsReducing legal liabilityMinimizing potential economic lossDecreasing potential exposure to disasterReducing the probability of a disaster occurrenceReducing disruption to normal operationsEnsuring organizational stabilityEnsuring orderly, systematic, and timely recovery
9
Disaster Planning StrategyDisaster Planning Strategy
Recovery Planning BenefitsMinimizing insurance premiumsReducing reliance on key individualsIncreasing asset protectionEnsuring the safety of personnel and patientsComplying with legal, statutory, and regulatory requirements
10
Disaster Planning StrategyDisaster Planning Strategy
Why have the stakes risen?eBusiness transitioned many businesses from 8am-5pm to a 24 x 7 x 365 model.Patient care could be compromised without information systems.Operations are running too lean to transition to manual processes and be able to conduct business ‘as usual’.Technology companies are not maintaining inventories as they once did to provide quick disaster shipment capabilities.New exposures: viruses, cyber-crime, terrorism
11
Disaster Planning StrategyDisaster Planning Strategy
Getting Approval & FundingHistorical data
The National Climactic Data Center (NCDC) is the “Nation’s Scorekeeper” in terms of addressing severe weather events in their historical perspective (www.noaa.gov)
National initiativesHospital Incident Emergency Command System (HIECS)
Regulatory audit complianceHIPAAJCAHO
12
Disaster Planning StrategyDisaster Planning Strategy
Building an Actionable Disaster Recovery Plan
13
Disaster Planning StrategyDisaster Planning Strategy
A Practical Approach
Detailed Assessment
Plan Development
Testing & Maintenance
Risk AssessmentInitiation
Phasing:1. Initiation 2. Risk Assessment3. Detailed Assessment4. Plan Development5. Testing & Maintenance
14
Disaster Planning StrategyDisaster Planning Strategy
Program InitiationInitiation
Detailed Assessment
Plan Development
Testing & Maintenance
Risk Assessment
15
Disaster Planning StrategyDisaster Planning Strategy
Strategic Objectives & ScopeObjective:
Develop overall Strategic Objectives and Scope for DRP Program
Practical Approach:Develop high-level Business Case to support DRP ProgramGather and review existing documentation related to DRP Identify areas of alignment with other Organization InitiativesDefine Program Objectives and Scope
Deliverables:DRP Program Definition
16
Disaster Planning StrategyDisaster Planning Strategy
Organizational StructureObjective:
Develop DRP Program Organizational Structure
Practical Approach:Identify Sponsorship, Stakeholders and Program ManagerDefine Program Organization, Roles and Responsibilities Dedicate existing Staff and supplement with External Resources
Deliverables:Identification of Sponsor(s), Stakeholders and Program ManagerDefinition of Program Organization, Roles and ResponsibilitiesInitial staffing of Core Team(s)
17
Disaster Planning StrategyDisaster Planning Strategy
Communication StrategyObjective:
Establish ongoing Communication Strategy
Practical Approach:Define Communication Objectives, Approach and Channels (e.g. Status Reports, Company Publications, etc.)For each Channel, define Audience, Message, Mechanism, Tactics, Measures and Timing Recommendations
Deliverables:DRP Communication Strategy and Timing Recommendations
18
Disaster Planning StrategyDisaster Planning Strategy
Program Plan & BudgetObjective:
Define High-Level DRP Program Plan and Budget
Practical Approach:Define and obtain consensus on Approach and Plan for the overall DRP ProgramEstimate DRP Program Cost and Resource Requirements
Deliverables:High-level DRP Approach, Plan and Budget Assessment
19
Disaster Planning StrategyDisaster Planning Strategy
Kick-Off MeetingObjective:
Facilitate Program Kick-Off Meeting
Practical Approach:Host Program Kick-Off Meeting, obtaining stakeholder consensus on Program Scope, Objectives, Communication Strategy, Plan and Budget
Deliverables:Program Kick-Off Meeting Presentation / AgendaKick-Off Meeting
20
Disaster Planning StrategyDisaster Planning Strategy
Risk AssessmentInitiation
Detailed Assessment
Testing & Maintenance
RiskAssessment
Plan Development
21
Disaster Planning StrategyDisaster Planning Strategy
Process Risk AnalysisObjective:
Perform Business Process Risk Analysis
Practical Approach:Interview Business and IT Subject Matter Experts (SMEs) to define disaster scenarios, create an inventory of the major business processes, define the impact of an interruption and thetolerance for downtime, and prioritize major business processesComplete Risk Assessment for Business Process
Deliverables:High-Level Business Process Current State DefinitionBusiness Process Risk Assessment
22
Disaster Planning StrategyDisaster Planning Strategy
Business Process Inventory
Application B0HResults to HISLabs
ADT, OrdersLab
Patient Care
Application A0H
Order requisition to ancillary systemOrdersADTOrder Entry
Patient Care
Applications Used
Downtime Tolerance
Impact of Interruption
Primary Outputs
Primary Processing
Primary Inputs
Business Function
Business Line
Process Dependencies
23
Disaster Planning StrategyDisaster Planning Strategy
Technology InventoryObjective:
Perform Technology Inventory and Risk Assessment
Practical Approach:Interview IT Subject Matter Experts (SMEs) to identify Technology Assets, define interdependencies and prioritize according to time sensitivity and criticalityAudit existing, relevant processes and proceduresComplete Risk Assessment for Technology Assets
Deliverables:High-Level Technology Asset Current State DefinitionTechnology Risk Assessment
24
Disaster Planning StrategyDisaster Planning Strategy
Technology Inventory
Technology Assets Quantity Location InterdependenciesDowntime Tolerance Criticality
Applications (1) Application 1Supported Desktops (1) Desktop config 1Networking Infrastructure (1) Network device 1PBX / Telephony (1) Telephony device 1
Total Valuation
25
Disaster Planning StrategyDisaster Planning Strategy
Detailed AssessmentInitiation
Detailed Assessment
Testing & Maintenance
RiskAssessment
Plan Development
26
Disaster Planning StrategyDisaster Planning Strategy
Detailed AssessmentObjective:
Perform Business Process Gap Analysis and identify Remediation Approaches
Practical Approach:Identify opportunities to prevent a disaster and other “quick-hits”Evaluate existing Policies, Workflow, and IT systemsComplete Detailed Current State DefinitionDefine and assess Remediation OptionsDevelop Recommendations and select Remediation SolutionDefine Future State based on implementation of selected SolutionDefine and obtain consensus on the Objectives, Scope, Approach Plan and Budget for Remediation Approach
27
Disaster Planning StrategyDisaster Planning Strategy
Detailed AssessmentDeliverables:
Detailed Current State DefinitionRemediation Options and Recommended Solution(s)Future State DefinitionRemediation Estimates and Plan
28
Disaster Planning StrategyDisaster Planning Strategy
Downtime Tolerance Costs
$$$$$$$$
Implementation Implementation Costs
$$$Costs
$$$
$$$$
$$
Downtime Tolerance (hours)Downtime Tolerance (hours)
1010 22 0024243636 44
29
Disaster Planning StrategyDisaster Planning Strategy
Disaster Recovery PlanningInitiation
Detailed Assessment
Testing & Maintenance
RiskAssessment
Plan Development
30
Disaster Planning StrategyDisaster Planning Strategy
Plan DevelopmentObjective:
Develop DRP plan
Practical Approach:Deploy “quick-hit” solutions Develop high-level recovery strategies and recovery phasesDefine roles and responsibilities including line of commandDefine disaster assessment and declaration definitions and proceduresDevelop emergency/ evacuation procedures that incorporate DRP activitiesDocument organization, staff and system functions and recovery requirements and procedures
31
Disaster Planning StrategyDisaster Planning Strategy
Plan DevelopmentEstablish recovery locations and document steps to make functional during a disaster Develop business partner and vendor agreementsDevelop communications plan and identify alternative communication toolsCreate contingency plans for missing people, failed proceduresDocument insurance information and proceduresBuild maintenance schedule and procedures
Deliverables:Actionable Disaster Recovery Plan
32
Disaster Planning StrategyDisaster Planning Strategy
Plan StructureSection 1: Plan Information
Objective: To provide information that will enable the reader or user of this plan to execute it while fully understanding the intentions and parameters with which it was created.
Content: Scope, Approach, Objectives, Team Organization, Pre-Disaster Action Checklist
33
Disaster Planning StrategyDisaster Planning Strategy
Plan StructureSection 2: Actionable Recovery Steps ***Disaster: Start
Here***
Objective: To provide a step by step checklist of activities that will be performed in the event of a disaster. This section contains the detail for each disaster level, by business line, by recovery option.
Content: Evacuation Checklist, Disaster Declaration Checklist, Recovery Team Activation Checklist, Level 1 – 4 Recovery Steps for all teams and for all recovery options
34
Disaster Planning StrategyDisaster Planning Strategy
Plan StructureSection 3: Addendums
Objective: To provide one place to access key information and resources required to efficiently and knowledgeably carry out the actionable recovery steps.
Content: Phone list, Insurance Information, Legal Considerations, Key Communication Messages, Facilities Considerations, Security Considerations, Transportation Options
35
Disaster Planning StrategyDisaster Planning Strategy
Section 1 - Plan InformationGoal:
Enable the user to execute the Plan while fully understanding the intentions and parameters with which it was created
Contents:ScopeApproachObjectivesTeam Organization Plan Activation ProcessDistributionCommunication StrategiesContingency Plans (missing people or failed procedures)
36
Disaster Planning StrategyDisaster Planning Strategy
Disaster Event Types
Event Types
Event Level 1 Event Level 2
Event Level 3 Event Level 4
37
Disaster Planning StrategyDisaster Planning Strategy
Recovery StrategiesEvent Types
Staff Facility TechnologyProcess
Event Level 1 Event Level 2
Event Level 3 Event Level 4
38
Disaster Planning StrategyDisaster Planning Strategy
Recovery Strategies
App 1 App 2 App 1 App 2 App 1 App 21 Execute manual procedures 1 1 1 1 2 Restore from backup 2 2 13 Failover to redundant systems 1 2
Event Level 1 Strategies
Busi
ness
Ar
ea 1
Busi
ness
Ar
ea 2
Busi
ness
Ar
ea 3
Event Level 1
App 1 App 2 App 1 App 2 App 1 App 21 Strategy 1 1 1 1 1 2 Strategy 2 2 2 13 Strategy 3 1 2
Event Level 2 Strategies
Busi
ness
Ar
ea 1
Busi
ness
Ar
ea 2
Busi
ness
Ar
ea 3
Event Level 2
39
Disaster Planning StrategyDisaster Planning Strategy
Recovery Team StructureStructured using the team approach
Each team has separate section of the Plan within each outage event levelRecovery teams = operational and technical groups responsible for restoring specific functionsEach team only has the authority to carry out the procedures contained in their section of the Plan
The teams are:Command TeamAdministrative Recovery TeamOperational Recovery TeamsTechnical Recovery Teams
40
Disaster Planning StrategyDisaster Planning Strategy
Recovery Team Structure
Command Team
Operational RecoveryTeam
Technical RecoveryTeam
Business Function 1 Business Function 2 Business Function 3 Phones Applications Infrastructure
Administrative Team
41
Disaster Planning StrategyDisaster Planning Strategy
Plan Activation Process
Authority to declare a disaster crucial element of plan:
Assigned to restricted number of individualsOnly group authorized to declare a disaster is the Command Team
Outage AlertOutage Alert
Command TeamCommand Team•• Establish command centerEstablish command center•• Determine disaster levelDetermine disaster level•• Supervise recovery stepsSupervise recovery steps
•• Receive initial alertReceive initial alert•• Determine disaster levelDetermine disaster level•• Activate recovery teamsActivate recovery teams
OperationalOperational TechnicalTechnicalAdministrativeAdministrative
•• Receive notificationReceive notification•• Evacuate areaEvacuate area•• Notify team membersNotify team members•• Activate planActivate plan
•• Receive notificationReceive notification•• Evacuate areaEvacuate area•• Notify team membersNotify team members•• Activate planActivate plan
•• Receive notificationReceive notification•• Evacuate areaEvacuate area•• Notify team membersNotify team members•• Activate planActivate plan
42
Disaster Planning StrategyDisaster Planning Strategy
Section 2 - Recovery StepsGoal:
To provide a step by step checklist of activities that will be performed in the event of a disaster. This section contains thedetail for each disaster level, by business line, by recovery option
In the event of a disaster, “start here”
Contents:Evacuation Checklist (OSHA)Recovery LocationsDisaster Declaration ChecklistRecovery Team Activation ChecklistLevel 1 – 4 Recovery Steps for Command, Administrative, Business Lines and Technical Recovery Teams and for all recovery options
43
Disaster Planning StrategyDisaster Planning Strategy
Command Team Checklist# Start
Day Start Time
End Day
End Time
Activity Team / Owner
Complete Date/Time
Comments
1. 1 E+ 00:00
1 E+ 00:15
Execute emergency response (fire, tornado, etc.)
ALL Refer to your facility emergency action plan
2. 1 E+ 00:15
1 E + 00:30
Determine the disaster level based on the Event Level Definitions below and proceed to Initiate Activation Checklist
Command
3. 1 E + 00:30
1 E+ 00:40
Notify Administration accordingly
Command
4.
1 E+ 00:40
1 E+ 01:00
Notify and activate the Recovery Team Leads what disaster level is being declared:
- Operational Team - Administrative Team - Technical Team
Command
44
Disaster Planning StrategyDisaster Planning Strategy
Command Team QuestionsGoal:
Remind staff about key action items that don’t necessarily belong in another checklist
Examples:Need Risk Management?Need Safety Team? Questions about safety procedures, personal injury.Need Purchasing Team? Need to purchase supplies, furnitrue, computers, etc.Need Facilities Team? Issues with HVAC, security, parking, restrooms, coffee?Need Communications Team? Issues with reporters, announcements, etc.
45
Disaster Planning StrategyDisaster Planning Strategy
Recovery LocationsDRP Locations
HospitalData CenterB900 - Basement
Who Goes Here?SysAdminsNetworkOPSTelecommunications
What Happens Here?Server / Network / Systems AssessmentsBackup restorationsFailover ActivitiesSystem Monitoring
Who Goes Here?DRP Command TeamCommunicationAdministrative Support
What Happens Here?Disaster Level DecisionsIssue ManagementActivity DirectionStatus ReportingCommunication
Command CenterConf. Room G & H
MedicalOffice Building
Resource CenterMOB 605b
Who Goes Here?Application SupportInterface TeamDBAs
What Happens Here?Application Assessments / RecoveryInterface Assessments / RecoveryDatabase Assessments / Recovery
Help Desk /Desktop ServicesMOB 215
Who Goes Here?Help DeskDesktop Services
What Happens Here?Help Desk 1st Level SupportDesktop Deployment / Support
46
Disaster Planning StrategyDisaster Planning Strategy
Section 3 - AddendumsGoal:
To provide one place to access key information and resources required to efficiently and knowledgeably carry out the actionable recovery steps
Contents:Phone List (staff, emergency contact, vendor)Insurance ChecklistTransportation ChecklistLegal ChecklistKey Communication MessagesSecurity Checklist
47
Disaster Planning StrategyDisaster Planning Strategy
Test Plan & ResultsInitiation
Detailed Assessment
Testing & Maintenance
RiskAssessment
Plan Development
48
Disaster Planning StrategyDisaster Planning Strategy
TestingObjective:
Perform testing
Practical Approach:Perform conference room test (passive testing)Perform full test (active testing)
Deliverables:Passive Test Plan and Test ResultsActive Test Plan and Test Results
49
Disaster Planning StrategyDisaster Planning Strategy
Plan StructureSection 1: Testing Plan Information
Objective: To provide information that will enable the reader or user of this plan to execute it while fully understanding the intentions and parameters with which it was created.
Content: Scope, Approach, Objectives, Roles and Responsibilities, Testing Environment and Locations, Assumptions, Known Risks and Issues
50
Disaster Planning StrategyDisaster Planning Strategy
Plan StructureSection 2: Actionable Testing Scenarios and Steps
Objective: To provide step by step conference room testing activities that address all levels of disasters that are represented in the Plan.
Content: Testing Checklists for Level 1 – 4 Disasters, Issue Management Process
51
Disaster Planning StrategyDisaster Planning Strategy
Plan StructureSection 3: Testing and Maintenance Schedule
Objective: To provide a schedule that will ensure that the Plan is tested and executed in a conference room setting at least two times per year. And, to develop a maintenance schedule that will ensure that the plan is current and relevant.
Content: Testing Activities and Schedule, Participants, Start Dates, End Dates; Maintenance Schedule, Owners, Due Dates
52
Disaster Planning StrategyDisaster Planning Strategy
Walk-Through TestIntended to orient and educated stakeholders with the organization and content of the PlanIntended to evaluate the Plan for completeness and accuracy, assuring all information is up-to-dateShould include all stakeholders of the BCP and take 1-2 hours to execute
Example: Walk-through Test Script
53
Disaster Planning StrategyDisaster Planning Strategy
Conference Room TestsObjectives:
Intended to evaluate the detailed checklists of the DRPBy creating scenarios (Level 1, 2, 3) to test different levels of the Plan, all stakeholders will have the opportunity to review individual checklists in addition to evaluating interdependencies between the checklistsShould include all stakeholders of the DRP and take 2-4 hours
Approach:For each level, develop a scenarioFor each scenario, define Type of Test, Participants, Type of Disaster, Day and Time of Disaster Event, Disaster Incident Description, Impact
54
Disaster Planning StrategyDisaster Planning Strategy
Discussion ItemsTo start the scenario
Who does what at that time?How long does it take?When is it finished?
What were the disaster event discovery procedures?What notifications need to occur?What documentation needs to be prepared?
How should a system outage be handled?What notification should occur?How do you validate the outage?How do you evaluate the impact on related systems?How do you document the process?
55
Disaster Planning StrategyDisaster Planning Strategy
Discussion ItemsWhat do you do in the meantime?How long do you continue manual processes? What if it is a hardware related problem? The vendor says it will be three days before it can be resolved. What do you do?
How long can they be used?Do they have adequate staff?How will they operate without access to the web?What should be communicated internally and externally?
What decisions need to be made and how quickly?Company personnel need to use their temporary operating procedures
What steps need to be taken?
56
Disaster Planning StrategyDisaster Planning Strategy
Active TestsIntended to evaluate the execution of the checklists and ensure everyone is comfortable executing their tasks
Should include all stakeholders of the DRP and take 4-8 hours
57
Disaster Planning StrategyDisaster Planning Strategy
Test ResultsImperative to track test problems in a Test Problem Log
Problem Number, Problem Description, Assigned To, Action Items
Intended to ensure action is taken on problems or issues that arose during the testing so that each iteration brings you closer to a complete plan
58
Disaster Planning StrategyDisaster Planning Strategy
Maintenance PlanInitiation
Detailed Assessment
Testing & Maintenance
RiskAssessment
Plan Development
59
Disaster Planning StrategyDisaster Planning Strategy
Ongoing MaintenanceObjective:
Insure DRP plans are maintained on an on-going basis
Practical Approach:Maintain DRP command team and recovery team rolesMaintain Vendor List and Supply ListPerform periodic Internal Audits/Reviews Insure change management processes incorporate DRP plan maintenance
Deliverables:Actionable DRP plans
60
Disaster Planning StrategyDisaster Planning Strategy
Timeline and Activities2 Weeks Prior to Test:
DRP Coordinator sends a message to all Command Team and Recovery Team Leads indicating the time of the testing and requesting Recovery Team Leads make checklist updatesRecovery Team Leads update checklists and distribute to BCP Coordinator
1 Week Prior to Test:DRP Coordinator updates BIA, Recovery Strategies, DRP and Test Plan
61
Disaster Planning StrategyDisaster Planning Strategy
Timeline and ActivitiesTesting:
Testing occurs over ½ dayDRP Coordinator facilitates all testing activities
3 Weeks After Test:Updates and other action items identified during testing complete
4 Weeks After Test:New DRP compiled and distributed to all Command Team and Recovery Team Leads and Executive Management
62
Disaster Planning StrategyDisaster Planning Strategy
Summary
63
Disaster Planning StrategyDisaster Planning Strategy
SummaryDisaster Recovery Planning is essential
Your approach needs to be practical and the plan needs to be executable
Test much and test often
Ensure the plan is maintained
64
Disaster Planning StrategyDisaster Planning Strategy
Jonathan ThompsonStoneBridge Group
701 Xenia Ave. South, Suite 170Minneapolis, MN 55416
(763) 923-7900(763) 923-7901 fax
www.stonebridgegroup.com