Pre-Disaster Recovery Plans

Presentation to:NACRC Legislative Conference

March 3, 2007Carol Foglesong, Orange County Comptroller’s

Office,Orlando, FL

andPaul Ketz, Broward County Records Division,

Ft. Lauderdale, FL

Agenda Introductions Purpose Unpredictability of Disasters Business Continuity Planning Definitions Reasons for a Pre-Need Contract What do you need to consider? Orange County’s Process to Contract Other Resources Questions

Purpose We are not here because we are

experts in the field of Business Continuity/Disaster Planning.

We are here because all of us are struggling with this topic.

We would like to share some of our experiences with you, and learn from the experiences of others here today.

Cycle of Disaster Planning

PREVENTION

RISK REDUCTION

MITIGATION

RECOVERYRESPONSE

PREPAREDNESS

So, what’s a disaster? Disasters come in all sizes and shapes

Big events: Hurricanes, Tornadoes, Earthquakes, Blizzards, Floods, Vandalism, Terrorism

Smaller events: Fires, Water Pipes, Roof Leaks

Disasters’ impact A large geographic area A localized area (county or city) A particular office operation/location

Hurricane Wilma Exposed office

suites are seen in the damaged Regents Bank Plaza on Tuesday, October 25, 2005 in Fort Lauderdale. (Sun-Sentinel/Joe Cavaretta)

Hurricane Wilma The Broward

County School Board building as seen from the 30th floor of the AutoNation Building in downtown Fort Lauderdale. (Brian Hekman, Coconut Creek)

Hurricane Wilma Virtually every

window on the west side of the Broward School Board building in downtown Fort Lauderdale was blown out by Hurricane Wilma. (Sun-Sentinel /Carl Seibert)

Hurricane Wilma Repairs are

underway at the Broward County Courthouse offices and parking garage in downtown Fort Lauderdale. (Sun-Sentinel.com)

What’s a common theme? Disasters are typically unexpected Disasters happen when you have

or are making other plans Disaster size or impact can’t be

readily predicted Disasters aren’t all BIG events; it

can be the little ones that cause the most concern and frustration

Disaster Readiness Business Continuity Plan

Staffing plans How to get up and running again

Disaster Recovery Plan Setting priorities on what has to be tackled first Papers versus computers

Forecast Disaster Plan Before the event planning Office preparations when a disaster is predicted

But who do you call for help? You’ve made a list

Businesses with addresses, names, phone numbers, faxes, and email addresses

Services offered and needed You’ve identified responsibilities of staff

Who calls whom when Who’s in charge

Do you and your staff keep a copy of your plan/phone lists with you? It won’t help if your list is at the office!

How to guarantee availability?

If a large area is impacted, there will be many organizations needing help; you may need to “get in line”

If a small area is impacted, will you get the specific help you need quickly?

If it’s a particular building or location impacted, what’s the best solution?

One answer A PRE-NEED CONTRACT A commitment to your organization

by a service provider You have a spot on the top of their

list You’ve pre-negotiated prices and

services You’ve already checked out who the

provider is and their references

Getting the Pre-Need Contract

You will have to convince your purchasing/contract group that a pre-need contract is not only possible, but logical

You knowingly commit to going through the purchasing process of your organization even with no known disaster on the horizon

Cost Be clear: no services = no dollars But money has to be available when

services are, in fact, necessary Does your organization require a “set aside”? Does your organization have a “reserve fund”?

Contract has to be renewed Annually? Every 3 years? What are your renewal options?

Who’s in charge of the procurement process? You may be the designated Records

Management Officer for your organization, but you will need: The purchasing/contract group Likely users must “buy in” Agreement on who sets the priorities for

service Recognition that the “disaster” may be small

or large scale Approval by your governing body

Who will negotiate the contract? Do they have a solid understanding of your business needs to secure the best vendor?

Other Considerations Can you opt for higher priority by

negotiating a retainer? Will the retainer be applied against services rendered?

If something happened to your materials, how long would it take to access them? Do you have 24/7 access? How do you get someone to the facility?

Is HIPAA a consideration in the custody/restoration of your records?

Response Time

Do you have a Continuity of Operations Plan (COOP)?

What kind of time will it take to respond to an event? For Emergency Services

Fire, Plumber, Repairmen For you and your staff For the Paper-Recovery Vendor

What events are importantbased on your geography?

Where are you located and what kinds of events do you expect every year? Florida – Hurricanes California – Earthquakes Northeast – Snow Low lying – Flooding What is it for you???

Vendor Capabilities How many linear feet of paper can be

recovered per hour per piece of equipment? Is the Vendor’s equipment wholly owned by

company, or is some subcontracted? Does the Vendor have a schedule of who

will respond after they are first notified? Process for chain of custody of records Where will the vendor set up “base camp”? What kind of turn-around time

Vendor Experience How much experience will you

require? What kinds of references do you

want to see? Past Performance

Program Name Owner / Architect Contract Amount / Percent Complete Completion Date

Be clear about the scope What media do you want used to

recreate your records? Film Electronic Back to paper?!?

Who can activate the contract Do you have other agencies/business units

who need access to this contract? If multiple agencies/business units can

access this contract, have you told the vendor who has priority?

Whose budget is used to pay for services?

Writing the RFP Plagiarize! Don’t reinvent the wheel

unnecessarily Each organization has their own

standard language and format Stick with the assistance particulars

that you might actually need Have a way to compare apples and

oranges between vendors/proposers

Computers vs Papers? Typically, there’s already a recovery plan

for computers, servers, databases, etc. Communication is key – talk to your IT

people and be sure they have a plan too! Computer recovery involves not only the

data, but also the equipment (machines, hardware)

There are specialized companies who handle this specialized service

Make sure your organization has an active and realistic plan in place

Paper or Paper Plus? Paper in a warehouse, awaiting its

retention time or its microfilming Paper in all those office filing cabinets

and individual desk drawers Microfilm? Microfiche? CDs? Maps or big drawings? Other media – besides computers?

Which documents first?

Are your documents prioritized? Do you store records for multiple

agencies? What gets saved first? Are your boxes segregated?

All things being equal, whose records are most important?

Does it make sense to secure your truly irreplaceable records elsewhere?

What type of paper recovery process will be used?

There are multiple types of document recovery/restoration techniques Vacuum Freeze Drying Molecular Sieve treatment Desiccant dehumidification HVAC decontamination Gamma Radiation (Remove Mold/Bacteria)

Are there techniques that cannot/should not be used on your records?

Paper Contamination Factors

Debris Sewage Silt Haz-Mat materials

Asbestos PCB Other petroleum products

Extras Re-filing, Re-indexing, Re-shelving Daily equipment rentals Small equipment purchases (shovels) Will they provide their own generators, or

does the vendor require electricity (water)?

Freight / Transportation Costs Travel / Per Diem or subsistence costs General Debris removal at your site to

gain building access Warranty on services

performed/document reproduction quality

Who does what? Inventory Packing Sorting/Identifying Prioritizing Labeling Loading/Unloading Re-shelving Security

Price per cubic foot

Pack out Gamma Radiation Cleaning Drying Ozone Deodorization

Price per each item Audio tapes Blueprints Maps Compact disks Diskettes Manuscripts Laser disks Photographs

Slides Microfiche Microfilm Negatives X-rays Pack out boxes Pack in boxes

Price per hour

Will any of the vendor’s services be provided per hour in consultation fashion?

Does the vendor have per hour charges for anything not specifically covered in your contract?

Where will the actual restoration and recovery take place?

Where will the vendor set up “base camp”? Onsite Remote Location Vendor’s Site

How will the materials be transported?

What kind of turn-around time does the vendor give for recovery onsite? Offsite?

Before you go back in… Who declares whether the building is

safe to re-enter? Fire Marshal Building Inspector Facilities Management Some other authority?

Where are your emergency triage supplies? What will you provide? What will the vendor provide? Where will the

vendor set up “base camp”?

Before you go back in…(cont’d) Will you need assistance to secure

the structure itself? Water removal Temporary Power Temporary building repairs

Securing Walls Roof Repairs

What is your plan if your staff balk at going back in?

One solution

Set up scenarios and make pricing those responses a part of the evaluation process

Pick 3 events of varying significance

Mix up the media to be recovered

Orange County FL Sample Full RFP and evaluation grading sheets

being lifted to NACRC document library Understand that we decided our biggest

problem was going to be water damage We don’t get snow or many tornadoes Substitute your own boiler plate front

end

Scope of Work Location for services (Orange County FL; mostly mid-

Orange County) Volume of work undetermined (who knows what’s going

to happen?) Not for computer systems What we asked for

There shall be no retainer paid in order to keep the contract in effect. The resulting contract will be on an as-needed basis and used only in case of a disaster or emergency as so deemed by the County. The Contractor shall provide 24 hours per day, 7 days per week emergency response service. Response by phone after the first notification shall be within 2 hours. The Contractor shall have a representative on site after first notification within six (6) hours with the necessary equipment to be transferred within twelve (12) hours.

Selection Criteria and PointsCRITERIA

Qualifications of StaffQualifications of FirmTechnical ApproachM/WBE UtilizationLocationFee Proposal

TOTALWelfare Recipient Hires

WEIGHT

10 20 30 10 (minimum allowed) 10 (minimum allowed) 20

100 5 bonus points

Qualifications of Staff “Include a listing of all staff to be assigned to provide

the required services and resumes for each describing experience, training and education in the required consulting services.”

Reasoning and/or recommended changes Tell them to make the bios BRIEF

You don’t need or want 5 pages per person! Clarify the kind of training or certification staff has

WHEN did they get that training or certification? “Identify staff experience working with governmental

entities and list those projects.” Reasoning and/or recommended changes

Hold the government entities and projects for the firm’s qualifications

You just get the same information in two places

Qualifications of Firm #1 “Provide a description and history of the firm focusing

on previous governmental experience. Provide the number of years in business showing proof of a minimum of 5 years in document recovery, the current number of full-time employees and the excess labor that can be provided in case of an emergency.”

Reasoning and/or recommended changes Does governmental experience actually matter? Five years in business seemed a logical threshold so that

we weren’t getting a firm that sprung up in response to our 2004 hurricanes

Full-time employees seemed important to evaluate whether the firm could respond to more than one disaster

Excess labor? Probably would just hire locally and teach/train while on the job

Qualifications of Firm #2 “Identify your equipment inventory available

to Orange County within the Southeastern United States.

Identify the capacity to process in linear feet per hour for each piece of equipment at each location.

Identify equipment wholly owned by your firm, or which is available through subcontracting.”

Reasoning and/or recommended changes Making up the equipment list was VERY interesting We decided on Southeastern US as a logical staging

area for us We learned you can lease/rent anything!

Qualifications of Firm #3 “List at least five references, with a minimum

of two from governmental entity experience, for which the firm has performed similar work including the contact name, address, telephone number, email, and date of the contract.”

Reasoning and/or recommended changes 5 references seems about the standard 2 government entities seemed to make purchasing

people happier Make sure to ask for email so you can readily contact

the references Send the reference organizations questions and then

call to interview them

Technical Approach #1 “Provide a brief description of the firm’s

approach to the project. Identify the equipment and recovery methodologies you employ to recover water-damaged paper documents, microfilm, CDs, maps and plans, etc.

Provide a schedule (not later than) of arrival of your equipment in hours after first notification.

Identify how you ensure records are properly inventoried and identified as well as the chain of custodianship.”

Technical Approach #2 “Identify the methods and instruments used

to assess moisture content of records. Provide an operational plan for rapid

deployment of required supervisors to provide direction to County employee “first responders” to assess damage and take interim steps to minimize loss. Include number of people, and number of hours for arrival.

Should no power be available, contractor shall be responsible for providing all power required to operate equipment.”

Technical Approach #3 “Provide a plan for how your clients are

prioritized in the event of a widespread emergency in which there is multiple-client demand.

Identify your permanent dehumidifying equipment locations in the Southeastern United States and the capacity in pounds of water extracted per hour for each piece of equipment at each location.

Present options available to provide dehumidifying of buildings and mold and mildew treatment.”

Technical Approach #4 “Submit your warranty or

guarantee of your services. Confirm the firm’s agreement to

meet the minimum requirements of this Request for Proposal.

Proposers may offer alternative solutions/options to achieve successful completion of the scope of work herein.”

Technical Approach Notes This area provided us with the greatest

flexibility in evaluating proposals and the largest points to award/withhold.

We don’t know if we got this area “right” as we haven’t had to use the contract.

But we did get it in writing in advance from the firm with which we contracted.

We drew upon notes from prior NACRC and disaster recovery sessions. We tried to imagine many/most scenarios.

Fee Schedule #1 “Submit a rate schedule for equipment, supplies, and

staff services. Travel, lodging, and meals shall be at rates allowed to

public employees under state law per state statute 112.061

Identify the cost per cubic foot, separately, for each of the elements as listed in item below. Provide a cost for all recovery services.

freeze conventionally blast freeze freeze dry rapid freeze treat for mold and mildew packing and unpacking treatment for soot”

Fee Schedule #2 “Identify mobilization and demobilization

rates for equipment and staff. Identify your markup for any equipment

rented, and supplies purchased. Identify freight cost per mile by type of

equipment. Provide the cost to implement the recovery

process. Provide the cost of the Recovery Service. Describe, if offered, your pre-registration

service and cost associated for such service.”

Fee Schedule #3 “In order to adequately evaluate and

compare services provided by proposers, three (3) hypothetical scenarios have been created for which proposers must provide a written response using prices submitted on Attachment A, the scope of work provided, and the technical approach of your firm. These scenarios are located in Attachment B.”

Scenario #1: Roof Leak It has just been discovered that the roof of the County’s

Records Center has been leaking for some time now. It is estimated from the buckling ceiling tiles that this has occurred over the past six (6) months.

When the actual damage was assessed, 200 boxes (dimensions: 10x12x15 = 1.2 cubic feet) containing paper documents and microfilm had been severely damaged by water, resulting in mold and mildew.

Contents of the boxes are as follows: 7/8 copier and bond paper and file folders, and 1/8 microfilm.

Based on the limited information in this scenario, the scope of work, and the services listed in Attachment A, the proposer is asked to provide estimated costs for complete recovery and restoration of the 240 cubic feet of documents. It is understood that the type or method of restoration will vary depending upon the type of records involved.

Scenario #2: Water Pipe Burst A water pipe has just burst in a County building,

which houses permits and application forms of the County. Standing water, approximately 8” deep has seeped into the records (i.e., CDs, microfilm, paper, maps, and plans, etc.). The damaged records consist of 1/2 copier and bond paper and file folders, 1/8 CDs, 1/8 microfilm, and 1/4 building maps and plans.

The room size for this County office is approximately 20,000 sq. ft. Housed in this office space are records contained in 10 four-drawer vertical letter-size filing cabinets (@ 6.0 cubic feet per unit for approximately 60.0 cubic feet total); 10 five-drawer lateral, letter-size filing cabinets (@ 13.0 cubic feet per unit) for approximately 130.0 cubic feet of records.

Scenario #2 (continued) This space also contains approximately 50

workstation cubicles at 100 sq. ft. each, which is equivalent to an estimated 4 to 5 cubic feet of records housed at each workstation. The 50 workstations contain approximately 250 cubic feet of water-damaged records.

Based on the limited information in this scenario, scope of work, and services listed in Attachment A, the proposer is asked to provide estimated costs for complete recovery and restoration of the records. It is understood that the type or method of restoration will vary depending upon the type of records involved.

Scenario #3: Small Fire A coffee pot with a potential electrical shortage problem

was left plugged into an outlet too long and sparked a small fire. Smoke from the fire triggered the sprinkler system, which caused water damage to some of the records.

When the actual damage was assessed, 300 boxes (dimensions: 10x12x15 = 1.2 cubic feet) containing paper documents, microfilm, CDs, and maps had been moderately damaged by both water from the sprinklers and soot from the fire. The damaged records consist of ¾ copier and bond paper and file folders, 1/8 CDs, and 1/8 microfilm.

Based on the limited information in this scenario, scope of work, and services listed in Attachment A, the proposer is asked to provide estimated costs for complete recovery and restoration of the records. It is understood that the type or method of restoration will vary depending upon the type of records involved.

Fee Schedule Notes

Having the scenarios was very important

We had trouble figuring out how to evaluate all the “standard” information the proposers submitted

The scenarios gave us a window into comparing apples to apples side-by-side

Checking References You really have to do this part We designed a series of 10

questions (lifted to the NACRC document library) which we sent to the reference people in advance of our call to them

We had at least 2 people and usually 3 people on each call – and then we compared notes

Doing the Evaluation We created spreadsheets and allocated

points among the criteria for each category (lifted to the NACRC Document Library)

Each of the evaluators completed their own spreadsheet separately

We then met together and compared scores; we were very close to each other

What one person couldn’t find or figure out, another person did

The Contract We selected our finalist just as

Hurricanes Katrina and Rita plowed in Getting our contract fully executed

dragged out for 3 months – and we only needed their properly executed insurance form!

Since the contract was finally and fully signed in January 2006, we have not had to use the contract

But we’re sure sleeping better!

Additional Thoughts

Start now; don’t wait for your “season”

Anticipate delays along the way From first concept to actual

completion was 2+ years for us Writing the scenarios and considering

all the possibilities requires multiple brains and various styles

Lessons Learned from Katrina Having a pre-need contract crosses the

“T’s” and dots the “I’s” in advance. Remember to look at the whole process New Orleans Parish experience with their

vendor, “The work that they did was excellent but the collection department was awful. They relied on a technical line in the work authorization to keep our books in Chicago until they were paid eliminating the opportunity for an audit of their bill. I suggest that any pre arrangements include reviewing the billing and collection methods.”

Use your NACRC Contacts Use NACRC as a resource to find

vendors Talk to other jurisdictions about

their experiences Conferences Listserv Email Conference Calls

Questions & Answers What experiences have you had? Good

or bad? When will you have a pre-need

contract? How will you develop your potential

proposer list to which to send the RFP? Can you join with other organizations in

your area? Can you “tag along” to an existing

contract someone else has?

Additional Resources NACRC www.nacrc.org Florida DOS State Library &

Archives http://dlis.dos.state.fl.us/disasterrecovery/#HVAC

National Parks Service – Wet Collections Recovery http://www.nps.gov/hfc/products/cons/wet-recovery.htm#

Thanks and for any follow-up Carol Foglesong, Assistant Comptroller,

Orange County Comptroller’s Office, PO Box 38, Orlando, FL 32802; 407.836.5982; carol.foglesong@occompt.com

Paul Ketz, Broward County Records Division, 115 S. Andrews Ave., Room 120,

Ft. Lauderdale, FL 33301; 954.357.7272; pketz@broward.org