Upload
camila
View
34
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Lessons Learned from Hurricane Katrina. Azim Ashraf Manager – Network Security & Incident Response. Personal Naiveté. Personal Preparations Some sense of excitement Estimation of what may occur Weather Channel – always on A bit of ‘Snow Day’ mentality. Hurricane Katrina. - PowerPoint PPT Presentation
Citation preview
Lessons Learned from Hurricane Katrina
Azim AshrafManager – Network Security & Incident Response
Personal Naiveté
• Personal Preparations
• Some sense of excitement
• Estimation of what may occur
• Weather Channel – always on
• A bit of ‘Snow Day’ mentality
04/20/23 LOUISIANA STATE UNIVERSITY 3
Hurricane Katrina
Thursday August 25
Sunday August 28
Tuesday August 23
Saturday August 27
Initial Projected Path
Monday, August 29 - Landfall• Katrina’s Immediate Effects
– Makes landfall 6:10 a.m.– Lower LA Parishes swamped by storm surge; no real word out– Parts of New Orleans flooded, at least one levee over-topped, but
city seems to have survived– SE Louisiana devastated by winds/rain– Mississippi seems hardest hit
• Monday 5pm Meeting at LSUPD Station – LSU is OK– LSU Survived … just a little damage on campus– Data Center Lost power but fail-over to back-up worked perfectly– Everything Looks “Good to Go” for Tuesday clean-up, Wednesday
start-up, and Thursday-as-usual– Mood lightened– Power restored to campus ~6:15pm
Tuesday 8/30 – Bad gets worse
• First confirmed reports of a levee failure in New Orleans occur at 1:30AM CDT
• By mid-day >80% of New Orleans is under water• Evacuees en route• LSU contacted about expanding routine special
evacuee facilities into a broader purpose– Medical Triage (Pete Maravich Assembly Center)– Special Needs Facility (Field House)– First IT needs – Phones, phones and more phones
Called to assist
• IT personnel needed to respond
• It was not going to be anything like a ‘snow day’
First Impressions
First Impressions
First Impressions
LSU – A city within a city
Large H. Ed. institutions uniquely positioned to respond• Infrastructure, knowledge, manpower, affiliations
– PMAC/Field House – Became the largest acute care hospital to date in in U.S. history
• Over 40,000 (?) patients processed during Hurricanes Katrina and Rita
– Established a Hurricane command center• Coordinated information for students, and evacuees, as well as directing
resources to where they were needed
– Faculty, staff, and student volunteers– Housing for responders– Crowd control– Food and laundry services– Long distance charges– Managed volunteers– Received and distributed donations
LSU – A city within a city (cont’d)– Tracked patients, volunteers, responders, supplies, etc..
– Provided Web page re-direction (and other IT services) for UNO
– Leveraged communications hardware and services to facilitate data or phone support for:
• Command centers
• Responders
• Govt. Agencies
• Affected Universities
• Evacuees
• Etc.
– LSU expended over $1M (not reimbursed)• Over $100K out of CIO’s budget
– LSU Became perhaps the most critical facility in support of disaster relief/response in the State of Louisiana
Lessons Learned at LSU• Buildings can be rebuilt; hardware can be
replaced. Data is the basis of continuity.• Knowing what you’ll need to do and having
it organized is more important than knowing exactly ‘how’ you’ll do it
• IT enables everything in the 21st Century• IT Personnel = First Responders
• Disaster Recovery and Business Continuity Planning is not a luxury
• Be prepared to be flexible; adapt, improvise, overcome
Lessons Learned at LSU (cont’d)
• Have a good stock of networking equipment, and mobile and desktop computing in the storeroom
• Have strong relationships with key vendors
• And most importantly…
People are your most key asset• Know who does what and have them ‘on reserve’• Expect them to be burdened with other priorities• Be prepared to be amazed…
Key changes in LSU’s Plan• Formal LSU EOC• Formal Memoranda
of Agreements (MOAs)– State agencies– Private sector
• diesel fuel from local refinery• water from local bottler, etc….
– Secondary suppliers backing up primaries• Chancellor requested written plans from all units
on campus• Full-time generator for PMAC• Logistics now pre-planned
Traditional Disaster Recovery- You’re down, everything else is fine
• Do you have a workable DR plan?
• Do you know where on campus you’ll go?
• Did you take necessary back-ups and do you have them ready to re-produce production files?
• What vendors will you need to tap – and for what?
• How will you quickly re-establish network connectivity? Phone service? Web presence? E-mail? Mission critical information systems?
Broader Disaster Recovery- You (and everyone around) you are down
• Are your off-sites conveniently (and perhaps tragically) close?
• Do you have arrangements to get key services restored at a distance– Web, E-mail, Financial/HR, Student Information, CMS
• Hot-sites may be too expensive – but can you find suitable raised floor/HVAC/power to ‘re-build’
• Can you support your administration “in exile?”– Internet access, computers, cell phones, e-mail, IM
• Is your ‘life-boat’ plan portable over larger distances?• Can you grab your key people? Can you care for them?
One Possible Tool In The Arsenal:Data Center Lifeboat• Situation: What if we had very short notice
(4-8 hours) notice of the need to abandon our data center/campus and set-up elsewhere (>50miles away)
• Goal #1: Re-establish some critical subset of services
• Goal #2: Support the re-establishment of some subset of university administration
Lifeboat• Key things to recover:
– Payroll/Financial Data– Web presence
• Splash/priority information screens
• As much content as possible
– E-mail service for faculty/staff/students
– Portal interface– Student Information Systems– HR, Procurement Systems– CMS– What else?
• Budgets ($25K, $50K, $100K)
• Key things to address– Off-site storage of critical back-ups– Ability to ‘grab and go’ key data
and hardware– List of key hardware needed later
from vendors– Disaster Supplies Crate
• What would we put into an 8x12 truck for rapid evac?
– Equipment for a mobile or relocated university command post
• Laptops, radios, phones, etc.– Identify Key IT personnel
• Who does what w/back-up• “Scoop ‘em up”
– Where might you go?
Survivor Disaster RecoveryYou’re the last ones standing
• Dealing with unimaginable demands– Start imagining it
• Do you have a stock of equipment to set up a large support operation in short-order?– Networking gear, computers, cables, supplies, telephone service
• Value of a flexible and capable staff• Consider how you’ll do all this on top of your normal
jobs, as campus life resumes and student enrollment increases
• How ready is your campus administration to take on the role of disaster response center?– Facilities, public safety/police, communications, academic affairs– Is the CEO (Chancellor, or President) prepared?
Final Thoughts
• Imagine the questions first so that you can find the answers
• Next time, you may not be watching it on CNN – you may be living it
• Do the right thing• Now is the time to think, plan, and take
action – later it will be too late
Final Thoughts
• Data is the basis of continuity• Have a flexible plan• People are your most key asset• Do the right thing because in
the end its really all about…
Service
Credits• The staff of LSU ITS who helped make the relief
effort a success.
• Brian Voss (CIO) – ‘In the Wake of Katrina’
• Brian Nichols (CISO) – ‘At Katrina’s Edge’
• Frank O’Quinn (DR) – ‘Weathering the Storm’
• Sheri Thompson, Jim Zietz, and others- photographs
• John Borne – excerpts from Master’s Thesis
• Margo Jolet, LSU Office of Public Affairs - ‘LSU in the Eye of The Storm’
Lessons Learned from Hurricane Katrina
Azim AshrafManager – Network Security & Incident Response