Upload
brett-jenkins
View
214
Download
1
Tags:
Embed Size (px)
Citation preview
1
Lessons Learned from the April 16, 2007 Tragedy
at Virginia Tech
EDUCAUSE Live!November 5, 2007
Erv Blythe, VP for Information Technologies
2
Presentation Topics
• April 16, 2007• The impact – load on the communications
infrastructure• Web Communications• The notification issue• Radio communications interoperability• Identity management and privacy• Data preservation and computer forensics• A new perspective on IT
3
April 16, 2007
• 49 people shot
• 32 dead plus shooter
• 9 minutes
4
Crisis Response
• About 1500 first responders• 27 ambulances from 14 agencies and five hospitals
5
Figure 1: Communications Infrastructure Stress Factors April 16, 2007 System Normal April 16 Effect
University Web Site Access 455 gigabytes per MONTH (largest ever)
432 gigabytes in a DAY
3000 % increase
Virginia Tech Police Dispatch Center
400 -500 calls per day
2,027 calls 45 0% increase
Cellular Provider Capacity and Coverage
Designed for non -emergency peak load, limited in -building coverage
Added 3 COLTs, 2 in -building antenna systems, 200 phones
By April 17, temporary coverage/capacity added
Internet gateway capacity 500 Mbps Added 1 Gbps over 10GE research link
300% increase
University Switchboard 3,200 calls handled per week
9,878 calls handled 4/16 -4/21
300% increase
Telephone calls into Blacksburg Central Office
Reported by local provider Multifold Increase
Virginia Tech Telephone System Inbound Calls
25,000 calls inbound daily on average
75,000+ calls inbound on April 16
300% increase
Centralized Computing Systems Data Storage
Prior to 4/16, roughly 300 Terabytes/day
Since 4/16, over 600 Terabytes/day
100% increase
Data Preservation (12 week period)
3,000 tapes 11,700 tapes 390% increase
Communications
6
A humongous traffic spike …
7
Response by Private Carriers
• After the initial response period on April 16, cellular providers including AT&T, Sprint-Nextel, Verizon Wireless, and US Cellular all dispatched technicians to increase tower capacity.
• By April 17, Sprint-Nextel, Verizon Wireless, and US Cellular each had “Cell on Light Truck” systems operating on campus and each had provided emergency-use phones and accessories.
8
Infrastructure Facilities Duties
During the crisis, Virginia Tech IT faculty and staff were called upon to:
• Install telephone and data communications for– At least 9 geographically dispersed command centers – As well as media workrooms and counseling centers
• Perform network traffic balancing under unusual load
• Begin data collection and preservation activities
• Assist in obtaining location
information or class roster information
9
Lessons Learned:
Communications Infrastructure
• Public safety access point forwarding• Information centers• Command and control centers• Language barriers
10
Web Communications
• Due to unprecedented load, the webpage had to be stripped down to bare essentials from the normal design to a “Lite” design
Lessons Learned:
11
Normal home page before 4/16
12
“Lite” home page on 4/16
13
Lessons Learned:
Radio Communications• Responders from several jurisdictions, all with
different radios• Radio transmission was less than ideal in some
locations• Deficiencies in interoperability and coverage of
police, fire, and rescue radio communications are decades-old problems in the United States.
14
Notification System
A significant challenge during an emergency is providing mass notification of a threat and instructions for response.
All of these methods were used on April 16:• Broadcast e-mail to @vt.edu addresses (via LISTSERV)• Broadcast voicemail to campus phones (offices and residence
halls)• Recorded message on the WeatherLine/Hotline• VT.edu (www.vt.edu) and the Virginia Tech News website• University switchboard• Public media (TV, radio, news websites)• Siren system
15
VT Alerts• A short list of vendors for this service
was identified prior to 4/16/07
• Post-4/16 the vendor review process was expedited
• 3n was selected to provide hosted services for sending emergency messages
– Through cellular phones– PDAs and other wireless devices– SMS/ text messaging– Email– Or to voice services
16
• Virginia Tech students, faculty, and staff can “opt in” to subscribe to an automated event notification system that will notify them of an emergency situation. Each subscriber can select up to three different methods to be contacted. These contact methods include:
• Text message (SMS) to mobile devices
• Instant message (AOL, MSN, ICQ, and Yahoo)
• E-mail (including non-Virginia Tech addresses)
• Phone call to office
• Phone call to residence
• Phone call to mobile number
• Phone call to another destination
VT Alerts Continued
17
QuestionsQuestions
18
Lessons Learned: Privacy
• Maintaining vigilant attention to the restrictions of FERPA and HIPAA added significant complexity to the urgent data needs of April 16th and the activities of the days that followed.
• Immediately following the tragedy, it was recognized that information concerning victims should be restricted from inclusion in the public directory. The University Registrar’s office was asked to mark the student records as “confidential” in Banner to ensure the information would not be publicly accessible from the Enterprise Directory.
• For selected individuals identified by university, state, or federal authorities, copies of the e-mail content, Filebox content, ePortfolio content, and voicemail recordings were preserved on compact disk (CD) or digital video disk (DVD.)
19
Lessons Learned:
Identity Management• Across many of the response activities, one
overarching difficulty from a data perspective was the consideration of the diverse constituencies.
• An integrated, enterprisecentric identity management system would have reduced some of the complexity of this process.
20
• Support for investigators
• Support for families and victims
• e-Discovery requirements
Data Preservation
21
QuestionsQuestions
22
DC ATLANTA
DIVERSE OPTICALCORE
ADVANCED IP NETWORK
INTEGRATEDAPPLICATIONS
Pervasive Computing
Massive Data Storage and Access
High Performance Computing
Integrated Services
Inherent Resilience, COOP
Anywhere, Any Device, Any Service
SERVICES & GLOBAL ACCESS
CyberSecurity
Integrated Emergency Applications
Sensor Networks
Strategic RecommendationIntegrated IP -Based
Information ArchitectureLegacy Systems
CENTRAL IT SERVICES
EDUCATIONAL SYSTEMS
CATV
DATA
TELEPHONE
911 DISPATCH
EMERGENCY RADIO
Near TermRecommendations
VT Alerts Notification
Phone Features
CATV Alert System
Crestron Control
WUVT Alert
Police Dispatch
PSAP Forwarding
Blackboard/Scholar
Improve Cellular
Update Personal Info
Location Awareness
Local Interop/Data
RESPONSE
RECOVERY MITIGATION
PREPAREDNESS
Role of IT inEmergency Management
DCDCDC ATLANTAATLANTAATLANTA
DIVERSE OPTICALCORE
ADVANCED IP NETWORK
INTEGRATEDAPPLICATIONS
DIVERSE OPTICALCORE
ADVANCED IP NETWORK
INTEGRATEDAPPLICATIONS
Pervasive Computing
Massive Data Storage and Access
High Performance Computing
Integrated Services
Inherent Resilience, COOP
Anywhere, Any Device, Any Service
SERVICES & GLOBAL ACCESSSERVICES & GLOBAL ACCESS
CyberSecurity
Integrated Emergency Applications
Sensor Networks
Strategic RecommendationIntegrated IP -Based
Information ArchitectureLegacy Systems
CENTRAL IT SERVICESCENTRAL IT SERVICES
EDUCATIONAL SYSTEMS
CATV
DATA
TELEPHONE
911 DISPATCH911 DISPATCH
EMERGENCY RADIOEMERGENCY RADIO
Near TermRecommendations
VT Alerts Notification
Phone Features
CATV Alert System
Crestron Control
WUVT Alert
Police Dispatch
PSAP Forwarding
Blackboard/Scholar
Improve Cellular
Update Personal Info
Location Awareness
Local Interop/Data
RESPONSE
RECOVERY MITIGATION
PREPAREDNESSRESPONSE
RECOVERY MITIGATION
PREPAREDNESS
RESPONSERECOVERY
RESPONSERECOVERYRole of IT in
Emergency Management
A New Perspective on IT
23
24
QuestionsQuestions
25
Other Resources
Governor’s Review Panel final report
http://www.vtreviewpanel.org/report/index.html
Information and Communications Infrastructure Group report
http://www.vtnews.vt.edu/documents/2007-08-22_communications_infrastructure.pdf