Disaster Recovery Planning …….Business Contingency Planning

A Business Model For Continuity Planning

David M. CrosbyInformation Assurance and Business

Sustainability

David M. Crosby

Former VP of Information Security, Venture Bank35 Years Experience in IT15 Years Experience in Information Security and Business SustainabilityFinance, Aerospace, Insurance and Energy Industry; and Technology and Services Company Principal

Introductions

Our World is Changing

HIPAAHIPAA

Int. AuditInt. Audit

Ext AuditExt Audit

State Regs.State Regs.

Disaster Recovery and Contingency

Operations Protect Information and

Processes

GLB NoticeGLB Notice

Institutional Best PracticesInstitutional

Best PracticesService To Our

CustomersService To Our

Customers

The Business Continuity Management Program

SB 1386SB 1386

County Regs.County Regs.

Federal Regs.Federal Regs.

The interruption of fundamental business processes for any extended period of time could have a debilitating

affect on our basic infrastructure…….and our way of life

E-Commerce

Private and Business Online Trading

Cash Advances At ATM Machines

Personal and Commercial Online Banking

Purchases By Credit Cards

Just In Time Inventories

Communications

Student Services

Grants and Endowments

General Administration & Finance

The Business Continuity Management Program

ERP – Emergency Response Plan: Steps Taken To Immediately Respond To An Event, Ensure Personnel Safety, Minimize Further Impact To Assets, And Make Proper Notifications.

DRP – Disaster Recovery Plan: Steps Taken To Restore Specified Infrastructure Requirements Such As Information Systems, Clinical Equipment Environments, Internal And External Network Connections, And Data Structures Utilizing Alternate Resources For Hardware, Software, Data, and Networks.

CMP – Crisis Management Plan: Steps Taken To Manage The Event To Ensure That Order Is Maintained, Employee Assistance Is Being Provided, Proper Information Is Being Disseminated By Appropriate Representatives, Action Items Are Effectively Escalated, And Ongoing Internal And External Notifications Are Consistent.

BCP – Business Contingency Plan: Steps Taken To Restore Alternate Business Processes In The Event That Automated Processes Or Business Infrastructures Are Unavailable, Employing Documented Workaround And/Or Manual Procedures And Alternate Resources.

ERP CMPBCPDRP

The Business Continuity Management Program

Working Components

ERP CMPBCPDRP

The Business Continuity Management Program

Response - Notifications, assessments, escalations, declarations, etc. (established procedures)

Recovery/Relocation - Mobilization, Quick-ship, Infrastructure, Network and Data recovery, etc.. Movement of staff, patients, and business units to alternate facilities (flexibility and adaptability)

Resumption - of Business Operations and I.T. functionality (business units must synch up processes and resume operations at an alternate site)

Re-assessment - of situation, strategies, planning, reactions (input from all involved parties)

Restoration - Movement back to home site and/or normal operations (reconstituted at restored site by I.T. and/or Business Units

Notification

Initial Notifications Telephone Trees

Command Center Assembly

Assessment and

Status

Damage AssessmentInitial Status ReportingSecondary Notifications

Organizational Committees Local Authorities Vendors Customers Media

EscalationsFirst Response

Personnel Safety Damage Mitigation Local Authorities

Evacuations

Components Of The Emergency Response PlanComponents Of The Emergency Response Plan

Declarations

ChecklistsScriptsProceduresContact Lists VendorsMobilization

Disaster Recovery Planning

Steps taken to restore specified infrastructure requirements such as Information Systems, business equipment environments, internal and external network connections, and data structures utilizing alternate

resources for hardware, software, data, and networks.

What To Do When The Computer Goes Down

Components Of The Disaster Recovery Plan

Disaster Recovery Is……

Flexible Response To A Crisis

Place to Recover (Location/Equipment/Network)

Defined “Recovery Set” (Critical Components)

Reliable Backups

Test – Maintain – Test

Service Continuation

The successful recovery of mission-critical I.T. services to the customer community in response to a crisis

Disaster Recovery is NOT…..Recovery of full environment

A business continuity plan

A replacement for conventional service plans

A trivial decision

Components Of The Disaster Recovery Plan

Applications Analysis

Questionnaires Interviews

Analysis Documented Profiles Test

Criteria/Objectives Recovery Plans

NetworkInfrastructure

Owned Equipment DR Vendor Equipment

Connectivity Requirements Test Criteria/Objectives

Remote Access Parameters Define ‘rogue’ FTPs

Identified Network Services

LDAP

DNS

Email

Intranet/Internet

Gateway Servers

Test Criteria/Objectives

Opens SystemsI.S.

Infrastructure

Hardware

Systems

Databases

TSO/CICS

Test Criteria/Objectives

Documentation

ChecklistsScriptsProceduresContact Lists Test Criteria/Objectives

Components Of The Disaster Recovery Plan

The period of time in which systems, applications, or I.T. functions must be recovered after an outage. RTO's are often used as the basis for the development of recovery strategies, and as a determinant as to whether or not to implement the recovery strategies during a disaster situation.

The point in time to which systems and data must be restored after an outage. RPO's are often used as the basis for the development of backup strategies, and as a determinant of the amount of data that may need to be recreated after the systems or functions have been recovered.

RECOVERY POINT OBJECTIVE: (RPO)

RECOVERY TIME OBJECTIVE: (RTO)

I.T. Requirements

Components Of The Disaster Recovery Plan

DRP – Disaster Recovery Plan: Steps taken to restore specified infrastructure requirements such as Information Systems, business equipment environments, internal and external network connections, and data structures utilizing alternate resources for hardware, software, data, and networks.

- Hardware - System Software- Data and Data Structures - Applications- Networks - Desktop Services- Production Support

DRP

BCP – Business Contingency Plan: Steps taken to restore alternate business processes in the event that automated processes or business infrastructures are unavailable, employing documented workaround and/or manual procedures and alternate resources.

- Relocation of Personnel- Availability of remote support services and network connections - Contingency office space

BCP

Components Of The Business Contingency Plan

Business Contingency Planning

Steps taken to restore alternate business processes in the event that automated processes or business infrastructures are

unavailable, employing documented workaround and/or manual procedures and alternate resources.

What To Do While The Computer Is Down

Components Of The Business Contingency Plan

Business Contingency Planning Is……

Flexible Response To A Crisis

Place to Initiate Contingency Operations (Systems/Network/Location/Personnel/Equipment)

Documented Systems Workaround Procedures

Alternate Resources

The successful response to an interruption in normal operating procedures and thus services to the customer community

Business Continuity is NOT…..Disaster Recovery, Emergency Preparedness, or Crisis

Management

A Permanent Solution

An I.T. Issue

Components Of The Business Contingency Plan

Alternate Resources

Personnel & Skill SetsFacilitiesVendors

Hardware/SoftwareCommunications

Documentation

ProceduresLogistical Support

FormsContact Lists

Mobilization

LogisticsLocation(s)

TransportationPersonnel

Alternate

Processes

I.T. WorkaroundsManual Business Processes

Alternate Data Capture

Business

Resumption

LogisticsTransition Back To I.T.

Validation/AuditNormal Operations

Business Cycles

Components Of The Business Contingency Plan

Business Continuity Planning Scenarios

Loss of I.T Services or Resources Loss of Functional Support Personnel Loss of Facility Loss of Network Connectivity Loss of Voice Communications Loss of 3rd Party Suppliers Loss of Business Partners

Components Of The Business Contingency Plan

Build Contingency Plans

Identify key functional components to establish the business environment

Define the alternate process requirements for each component

Ensure interdependent business processes are identified and can be synched up

Define minimal processing requirements for each component

TEST - TEST - TEST - TEST

Components Of The Business Contingency Plan

Business Recovery Requirements

When do I have to have an alternate process in place to address loss of primary functions (I.T. and otherwise) ?

RECOVERY POINT OBJECTIVE: (RPO)

RECOVERY TIME OBJECTIVE: (RTO)

How current does my information have to be when normal processes are resumed ?

Components Of The Business Contingency Plan

Centralized Administration and Coordination Decentralized Development, Maintenance and Execution

Web-Enabled – 24 x 7 x 365 access from anywhere with VPN connection

Automated progress reporting during Plans development, maintenance, and execution

Define relationship between BCPs and DRPs (RTO and RPO)

Capable of expanding to include ERP and CMP

Real-time updating to a single database, not multiple Plans

Version Control on all Plans

Concurrent Plan development

Issue Templates

Import Templates

Develop BCPs

Flexibility when producing BCPs…………..or executing BCPs

“Show me all Plans by Department….”

“Show me all Plans by Building…..”

“Show me all Plans by Building, by Floor…..”

“Show me all Plans by Building, by Floor, by Department

Components Of The Business Contingency Plan

Negotiate The Service Level Agreement Between I.T. And Business Operations

Use Both The I.T. And Business RTO & RPO As The Basis

Disaster Recovery Plan Test Results Quantify Timelines

Business Contingency Plan Exercises Qualify Impact

I.T. Capabilities Improve Timelines – But At A Cost

Business Contingencies Reduce Impact - But Require I.T. Capabilities

Criticality Rankings

Systems Recovery Sequencing

Business Process Prioritization

I.T. and Business Process Timelines

Negotiated RTO and RPO

Components Of The Business Contingency Plan

Results

I.T. Better Understands The Customers’ Issues and Requirements

I.T. Obtains A Clearly Documented Set Of Customer Expectations For DRP’s

- Clarify and Justify Budget Forecasts

- Establishes Specific Test Objectives

- Ensure Active Customer Involvement In Testing & Recovery Processes

Business Units Better Understand The Role Of I.T. In The Contingency Process

Business Units Obtain A Set Of Parameters From Which To Develop their BCP’s

- Workaround Procedures During Downtime

- Procedures For Capturing Lost Transactions From Downtime and During Recovery

- Restoration Of Normal Environments

Components Of The Business Contingency Plan

EventAnalysis

Catastrophic Events

Criminal Events

Disease/Epidemics

Technological or Safety

Utility or Structural

Weather

Personal vs. Professional

Reaction Planning

Local MediaEmployees

Local AuthoritiesOpennessAccuracyBalance

Designate a point person

Continuous Flow

Communications

Emotional Assistance

Addressing Traumatic Stress

Family Assistance Pgms

Professional Assistance

Provide Information & Counseling

Post Incident Follow-up

Components Of The Crisis Management PlanComponents Of The Crisis Management Plan

Documentation

Employee Checklists

And Action Plans

Press Release Data

Employee Notification Mechanisms

1. Identification of vulnerabilities

2. Performance of regional threat assessment

3. Assessment of system resources

4. Communications infrastructure

5. Standardization of plans

6. Dissemination of information

7. Analysis of system Surge Capacity

8. Collaboration with federal, state, local agencies

Crisis Management PreparednessKey Elements

Components Of The Crisis Management PlanComponents Of The Crisis Management Plan

Regional Collaboration

Local Fire/EMS/OES Law Enforcement Health Dept./Hazmat Hospitals

State State Health Dept. State OES/DHS Hospitals

Federal Federal Emergency Mgmt Agency CDC Military

Private Sector Collaboration Individual Plans

Supplement/Complement Broader Plans

Clinical Care Response Public Health Response

Who does what?? Who calls whom??

Components Of The Crisis Management PlanComponents Of The Crisis Management Plan

When the issues surrounding both I.T. Disaster Recovery Plans and Business Unit Business Contingency Plans come together what is at stake becomes much clearer, and each can understand the others objectives and expectations. Only then can a total Business Continuation Program be effective.

And if the organization has an effective Business Continuation Program, not only can it assure that its goals and objectives will be met…..but will also become a valued partner in the protection of the larger infrastructure.….

The Business Continuity Management Program

Helping Others