53
1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

Embed Size (px)

Citation preview

Page 1: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

1

The National Finance Center and

Hurricane Katrina

John White, Deputy Director

June 2006

Page 2: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

2

USDA, Office of the Chief Financial Officer

Chief Financial Officer’s View - USDA Characteristics:

- 110,000 employees @ 14,000 offices and field locations- 300+ national programs - $77B in annual spending- $128B in assets- $100B in loans

Program Management Responsibility Includes:- Financial leadership across the enterprise; roughly

equivalent in size to 6th largest private sector firm in U.S.- Financial policy and planning- Financial systems and operations- Government-wide service delivery through the National

Finance Center (NFC)

Page 3: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

3

Scope of USDA CFO Operations inNew Orleans (co-located)

National Finance Center (NFC)- Government-wide payroll/personnel for 585,000 employees- Government-wide Human Resource services- Government-wide Health Benefits Programs- Disbursements and Collections- Data Center Services- At time of Katrina, over 1300 employees

CFO Controller Operations Division- Provides financial services to USDA agencies, including administrative payments

(2.5 million annually), interagency payments, reconciliations, and vendor file- Assembles USDA financial reports, including FACTS I & II, and annual consolidated

financial statements- At time of Katrina, almost 300 employees

CFO Financial Systems Division- Operates the corporate consolidated financial system for all USDA agencies- Operates administrative systems, such as travel and property, for all USDA agencies- Creates and reconciles data extracts for consolidated audit and financial statements- At time of Katrina, over 70 employees

Page 4: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

4

Prolog: Accountability – Guiding Principles

Embrace that you are in charge and responsible for your organization, its finances, and people.

Be prepared to go it alone, do what is necessary to get the job done and care for your staff. In a major situation, there may be no one else to consult about how to proceed.

Never forget that during a crisis, people demand leadership, decisions must be made--often in a vacuum, stick to the plan but be flexible.

Page 5: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

5

Preface: Risk Management – Guiding Actions

Managing to deliver the expected business results by considering risks in doing what we do.

Focusing on most significant risks associated with the nature of the business:- Economic – controls & testing based on what is at stake- Operational – environmental, internal, natural, compliance,

technical, organizational, and human risks

Keys to successful risk management:- Understand the risk profile of the business- Leadership sets the tone- Integrate risk management into day-to-day practices and

decisions- Evaluate on clarity, transparency, integrity, & performance

Page 6: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

6

Preface: Business ContinuityApproach – Prepare!

Business Impact Assessment Disaster Recovery Plan Business Continuity Plan Uninterruptible Power Supply Emergency Power Supply Two annual NFC drills USDA and Government wide drills Three historical near misses

Page 7: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

7

Page 8: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

8

Hurricane Katrina

Activated DR on Friday

evening Deployed Advanced Teams

on Saturday morning Completed payroll late

Saturday evening Shutdown data center and

facility early Sunday Made disaster declaration

Sunday evening Began implementation of

COOP on Monday

Initial Timeline:

Page 9: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

9

COOP Concept

Subscription service for

data center, workstations, and bulk print and mail

Philadelphia site used for data center and 101 data center staff seats

Grand Prairie site used for 150 business operations seats

Plans all geared to loss of New Orleans facilities

Concept of operations:

Page 10: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

10

Page 11: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

11

Realities of Katrina

Implications were larger

than anyone imagined This would be a long-term

situation Immediate focus was:

- Service to the customers- Accountability and

caring for employees- Fiscal accountability

Balance delivery and personal needs

Initial reactions:

Page 12: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

12

Oversight Priorities

Incorporating local efforts

into Departmental and Government-wide efforts

Keeping long-term, big picture in focus

Finding ways to add value while allowing those on the line to keep things moving: - Policies- Resources- Contracting- Financial- Liaison - Monitoring and Reporting

View from the top:

Page 13: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

13

Locating and Deploying Staff

Local telecommunications

infrastructure mostly inoperable

Pre-planned out-of-town contact numbers functioning

No single data base for locating people in shelters

Travel impacted by roads and gas shortages

Family issues had to be addressed

First 14 days:

Page 14: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

14

Expanding Capabilities to Meet Long-Term Deployment

Duration of COOP means

more seats needed- Trailers acquired in Atlanta- USDA sites used in D.C.,

Kansas City, Roslyn, and Alexandria

- Customer sites in Birmingham and Fairfax leveraged

Secure connectivity and postal mail were two key infrastructure challengesto achieve

First 14 days:

Page 15: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

15

First Job, First

Critical data center resources are recovered

Essential services are restored

Payroll is made timely while migrating 60,000 new payees (565,000 total)

Financial systems returned online

First 14 days:

Page 16: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

16

Settling In for the Long Haul

Subscription service

requires sharing of resources during 6-week recovery period

Essential services require dedicated resources

Impact of Katrina implies 6 months not 6 weeks

Customer dedicated connectivity key issue

Decision made to buildout in “cold site”

First 14 days:

Page 17: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

17

Focus on Service Delivery

Resuming all services

requires that all resources (human, technical, and other) be available

Service delivery focus isfor Agency customers, payees, vendors, and employees

Give employees the challenge and the toolsand stay out of their way

First 14 days:

Page 18: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

18

New Orleans Critical to a Timely Return to Full Service

Returning to full service

required a full staff Deployed status had limited

seat availability, some security concerns, was costly, and came at a personal sacrifice to many

The New Orleans facility has the infrastructure to address business needs and support the additional population

Next 60 Days:

Page 19: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

19

Reconstitution in New Orleans

Housing, family issues, telecommunications,postal services, and local infrastructure had to be addressed- Security- Logistics- Habitation necessities- Health care

Family & friends, FEMA trailers, and cruise ships address most housingneeds

Next 60 Days:

Page 20: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

20

It’s Nice to Have Friends

USDA Secretary Johanns

and entire USDA family strongly supported COOP needs and the employees

Tremendous outreach by Federal colleagues, local Government, local organizations, and peoplein deployed locations

Food, clothing, education opportunities, friendship, and other assistance

Keys to success:

Page 21: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

21

It’s Nice to Be Prepared

Managers and involved

staff built the plan Same managers and staff

practiced the plan Live and desktop drills

under varying scenarios honed the mental and technical readiness

We know our customers rely on our mission delivery in order to sustain theirs

Keys to success:

Page 22: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

22

Lessons Learned – Governmentwide

Consolidated information

source needed to support decision-making

Central clearinghouse for needs and resources

Designation of essential services and associated rights is needed

Handbook to guide Federal Agencies through extraordinary authorities

Single coordination point for Federal efforts andissues

Learned:

Page 23: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

23

Lessons Learned - NFC

Communications are

always the challenge; planned communications channels are a real aid

Having a well-drilled plan allows for the basics to be accomplished with little intervention and allows management to focus on exceptions and surprises

Learned:

Page 24: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

24

Subscription service not

an ideal model for essential service provider

Administrative support a key aspect of support for long-term deployed staff

For long-term deployment, coordination of outreach efforts targeting employees becomes an essential task

Lessons Learned - NFCLearned:

Page 25: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

25

Some organizations

benefit from close proximity with one another

Must revalidate plan assumptions, plan contents, and business requirements annually

You cannot assume infrastructure items will be there for you

Lessons Learned - NFCLearned:

Page 26: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

26

Working to Mitigate Future Risks

Data Center equipment will

remain at subscription service location until new primary facility is selected

New AWS alternatives being explored

All business operations are being reassessed based on experiences

Lessons learned are being incorporated into revised DR and COOP plans

Page 27: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

27

People, Planning, & Practice

People who understand &

believe in DR and COOP make it happen

Plans have to account for your threats, the business requirements for COOP, and the people side of things

Practice is the best teacher. Practice on a regular schedule (top priority) and against changing scenarios. Capture the valuable lessons from each exercise.

Page 28: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

28

Epilog

Facing the most devastating

natural disaster to impact the United States, the employees of NFC set aside their personal concerns and focused on delivering for their customers.

Given such an event, the best came out of people, far and wide, and we are all better because of it.

Page 29: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

29

One-Year Review of Key Indicators

Housing rehabilitation and demolition are well underway while the housing market tightens- Pace of demolitions has accelerated in the last six months- Number of permits issued for rehab has nearly doubled- Rent prices have increased by 39% and home sale prices

have spiked in suburban parishes

Public services and infrastructure remain thin and slow to rebound- Approximately half of the bus and streetcar routes running

with only 17% of buses in use- Gas service reaching 41%; electricity reaching 60% of pre-

Katrina customer base

Page 30: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

30

One-Year Review of Key Indicators

Labor force in the region is 30% smaller than one year agowith slow growth over the last six months- Unemployment rate remains higher than pre-Katrina- The New Orleans metro area lost 190,000 workers- In the past six months, 3.4% more jobs but much of that may reflect the rise

in new job seekers- Unemployment rate is now 7.2%, higher than last August

Over $100 billion in federal aid dedicated to serving familiesand communities impacted by hurricanes Katrina, Rita,and Wilma

Number of displaced and unemployed workers remains high- Federal government approved approximately $109 billion in federal aid to the

Gulf Coast states- Half has been dedicated to emergency and longer-term housing- Estimated 278,000 workers are still displaced- 23% remain unemployed

Page 31: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

31

Page 32: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

32

Page 33: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

33

Page 34: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

34

Page 35: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

35

Page 36: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

36

Page 37: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

37

Page 38: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

38

Page 39: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

39

Page 40: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

40

Questions???

Please ask us questions important to you.

“Lagniappe” is a New Orleans’ term often used to indicate getting something extra in a

transaction. Our lagniappe for you today includes additional pictures of the post-Katrina

New Orleans area.

Page 41: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

41

LagniappeCanal Street in CBD

Page 42: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

42

LagniappeLower 9th Ward

Page 43: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

43

LagniappeMold

Page 44: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

44

LagniappeThe lake is on the other side of the building

LagniappeThe lake is on the other side of the building

Page 45: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

45

LagniappeBoat came 3 miles inland

LagniappeBoat came 3 miles inland

Page 46: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

46

LagniappeBefore

NATIONAL FINANCE CENTER

LagniappeBefore

Page 47: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

47

LagniappeAfter

LagniappeAfter

Page 48: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

48

LagniappeAnyone missing a hot tub?

LagniappeAnyone missing a hot tub?

Page 49: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

49

LagniappeA quite country road

LagniappeA quiet country road

Page 50: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

50

LagniappeMain bridge into Eastern New Orleans

LagniappeMain bridge into Eastern New Orleans

Page 51: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

51

LagniappeA view from the elevated expressway

LagniappeA view from the elevated expressway

Page 52: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

52

LagniappeBourbon Street – Never so clean!

LagniappeBourbon Street – Never so clean!

Page 53: 1 The National Finance Center and Hurricane Katrina John White, Deputy Director June 2006

53

NSAA IT Conference