HSEEP ExerciseEvaluation and Improvement
ODP’s Mission
Primary responsibility within the executive branch to build and sustain the preparedness of the US to reduce vulnerabilities, prevent, respond to, and recover from acts of terrorism (Homeland Security Act).
ODP’s Responsibilities
Grant programs for planning, equipment, training and exercises
National training program
National exercise program
Grant Programs
State Homeland Security Program
Law Enforcement Terrorism Prevention
Program
Citizen Corps Program
Urban Areas Security Initiative Program
Fire Fighter Assistance Program
State Homeland Security Program
Purpose: to enhance capacity of states and locals to prevent, respond to, and recover from terrorism
Provides funds for Homeland security and emergency operations
planning The purchase of specialized equipment CBRNE and cyber security training programs CBRNE and cyber security exercises State Homeland Security Assessments and
Strategies
Law Enforcement Terrorism Prevention Program
Provide law enforcement communities with funds to support the following prevention activities:
Information sharing to preempt terrorist attacks Target hardening Recognition of potential or developing threats Interoperable communications Intervention of terrorists before they can execute a
threat Planning, organization, training, exercises, and
equipment
Citizen Corps Program
Provides funds to support Citizen Corps Councils with planning, outreach, and management of Citizen Corps program and activities
Form and sustain a Citizen Corp Council Engage citizens in homeland security Conduct public education and outreach Develop and implement Citizen Corps programs Coordinate Citizen Corps activities with other DHS
funded programs and other federal initiatives
Urban Areas Security Initiative Program
Address the unique needs of large urban areas – 50 cities
Conduct jurisdictional assessment and develop Urban Area Homeland Security Strategy.
Funds for planning, equipment, training, exercise, and administration and operational activities related to heightened threat levels
Fire Fighter Assistance
Protect public and fire fighters against fire and fire-related hazards Fire fighting Operations and Safety Fire Prevention Fire fighting Vehicles
Strategy Process Overview
END RESULTEND RESULT = Capability Improvements
Statewide HomelandStatewide HomelandSecurity StrategySecurity Strategy
Conducted at the local and state levels
Created at the regional and state level
Created by ODP
AssessmentsAssessments State AssistanceState AssistancePlanPlan
State and Urban Area use strategy to identify & allocate all HS resources
Public Health
Health Care
Public Works
Government Administrative
Private Sector
Non-Profit/Voluntary Sector
State and local jurisdictions All First Responder Disciplines
Strategy Participants
Fire Service
HazMat
Emergency Medical Services
Law Enforcement
Emergency Management
Public Safety
Communications
Assessment Overview
Shortfalls or “Gaps”
Agricultural Vulnerability Assessment
* CBRNE: Chemical, Biological, Radiological, Nuclear, Explosive
ThreatAssessment
VulnerabilityAssessment
Planning Factors
CBRNE* Scenarios
RequiredCapabilities
CurrentCapabilities
Risk Assessment Needs Assessment
StatewideHomelandSecurityStrategy
Threat Assessment
Agricultural Vulnerability Assessment
Risk Assessment
ThreatAssessment
VulnerabilityAssessment
Who:
Local, state, and federal law enforcement officials
What: • Identify number of Potential
Threat Elements (PTEs)
• Identify threat factors (existence, violent history, intentions, WMD capability, and targeting)
• Identify motivations (political, religious, environmental, racial, or special interest)
• Identify WMD capabilities (CBRNE)
Vulnerability Assessment
Risk Assessment
ThreatAssessment
Who:
All response disciplines at local, state, and federal levels
What:Identify critical infrastructure/ potential targets
Evaluate targets for:
• Level of visibility
• Criticality of target site
• Impact outside of jurisdiction
• Access to target
• Target threat of hazard
• Target site population capacity
• Potential collateral mass casualties
Agricultural Vulnerability Assessment
VulnerabilityAssessment
The results from the risk assessment process (threat and vulnerability) provide a link to the capabilities and needs assessment process.PlanningOrganizationEquipmentTrainingExercises
Capabilities and Needs: Planning
State Homeland SecurityStrategy
Developed by State based on local needs
Provides blueprint for planning of homeland security efforts to enhance preparedness and for use of resources
StateAssistance Plans
ODP uses the strategies and needs assessment data to tailor and formulate a State/Metro Assistance Plan (SAP/MAP) for each state
A SAP/MAP is a blueprint for the delivery of ODP training, exercise, technical assistance and equipment services
National Training Program
Training for federal, state and local homeland security professionals
Based on critical tasks to prevent, respond to or recover from a terrorist incident
Over 40 courses available
ODP Training Program
ODP offers more than 40 courses (Examples)
Live chemical agents training – Center for Domestic Preparedness
Live explosives training – New Mexico Institute of Mining and Technology
Radiological and nuclear agents training – Nevada Test Site
Advanced emergency medical training using human patient simulators – Texas A&M
Training on bioterrorism – Louisiana State University
National Exercise Program
Responsible for National Exercise Program
Threat and performance-based excises at federal, state, local, and international levels
Strategy and Exercise Planning Workshops to define exercise needs and plan for each state
Assess Program Success Through Exercises
Performance measures for ODP’s grant, training, and exercise programs are tied to performance of critical tasks
Percent of jurisdictions that can perform critical tasks as demonstrated through exercises 500,000+ population 100,000+ population 50,000+ population
Overview of HSEEP
Threat- and Performance-based Exercises
Cycle of exercises
Increasing complexity
To improve preparedness
HSEEP Manuals
Volume I: Program Overview and Doctrine
Volume II: Exercise Evaluation and Improvement
Volume III: Exercise Development Volume IV: Sample Exercise
Documents and Formats
Vol I: HSEEP Overview and Doctrine
ODP’s exercise and evaluation doctrine Uniform approach for exercise design,
development, conduct, and evaluation Exercise design and implementation
process Suite of common scenarios (TBD)
Vol II: Exercise Evaluation and Improvement
Defines exercise evaluation and improvement process
Provides uniform set of evaluation guides
Defines data analysis process Includes standardized After-Action
Report template
Vol III: Exercise Development
Defines exercise planning and design process
Provides guidance for the development and conduct of various types of exercises
Vol IV: Sample Documents
Provides sample letters, planning documents, checklists, scenarios, etc.
Reduces development time for exercise design team
Exercise Evaluation
Assess preparedness at federal, state and local levels
Validate strengths and identify improvement opportunities, resulting in improved preparedness
Provide guide for resource allocations
Evaluation Enhancements
Focus on performance of critical tasks and mission outcomes
Use of uniform evaluation tools Enhanced data analysis Debriefing meeting with key officials Improvement Plan Track implementation of improvements Suite of common scenarios (TBD)
Exercise Evaluation Methodology Development
Exercise Evaluation Working Group
Builds on Responder Guidelines
ODP exercise experience
CSEP and other programs
Will continue to enhance and improve
Exercise Evaluation and Improvement Process
Exercise Evaluation and Improvement Process Data Collection and Analysis
Step 1
Plan & Organize the Evaluation
Step 1
Plan & Organize the Evaluation
Step 2
Observe the Exercise &
Collect Data
Step 2
Observe the Exercise &
Collect Data
Step 3
Analyze Data
Step 3
Analyze Data
Step 4
Develop After Action Report
Step 4
Develop After Action Report
Improving Preparedness
Step 5
Conduct Debriefing
Step 5
Conduct Debriefing
Step 6
Identify Improvements
Step 6
Identify Improvements
Step 7
Finalize After Action Report
Step 7
Finalize After Action Report
Step 8
Track Implementation
Step 8
Track Implementation
Evaluation Planning, Observation, and Analysis
Levels of Analysis
Performance is analyzed at three levels: Task level Agency/discipline/function level Mission level (within and across
communities)
Levels of Analysis
Task Level Performance Answers the question: did
the person or team do the right thing the right way at the right time?
Helps assess need for training, equipment, personnel, etc.
Task = work with measurable output that has utility
Levels of Analysis
Agency/Discipline/Function Level Performance — Multiple teams Answers the question: did the larger team or
organization perform duties in accordance with plans and policies?
Helps assess communication, coordination, planning budgets, etc.
Levels of Analysis
Mission Level Performance Answers the question: were the
mission level outcomes achieved?
Addresses jurisdictional preparedness
Outcomes = results
Mission Outcomes
Pre-Event
Emergency Response
Post-Event
Prevention/Deterrence
Emergency Assessment Emergency Management Hazard Mitigation Public Protection Victim Care Investigation/Apprehension
Recovery/Remediation
Evaluation Requirements
Determine what outcomes will be evaluated, based on exercise objectives
Identify activities to be evaluated Identify which functions should be
observed Determine where observations will take
place Identify the appropriate evaluation tools
Exercise Evaluation Guides
ODP has developed Exercise Evaluation Guides that: Identify the activities that the evaluator
should be observing Provide consistency in tasks across
exercises Link individual tasks to disciplines and
outcomes
The EVALPLAN
Exercise-specific information
Plans, policies, procedures, and agreements
Evaluator recruiting and assignments
Evaluator training and instructions
Recruiting and Assigning Evaluators
Setting expectations – evaluators must be available for: pre-exercise training and briefing pre-exercise site visit the entire exercise (hours to days) post-exercise hot-wash post-exercise data analysis (1 day) contribution to the draft AAR
Recording Observations
The emphasis is on Who? What? When? Where? How? Why?
Record observations through: use of Evaluator Guides blank sheets of paper
Collect exercise documents
Record Significant Activities
Initiating scenario events Facility activities Response actions Key decisions made by Players Deviations from plans and procedures Completion time of events
Evaluator Summary
Compile observations into chronological narrative of events
Describe outcomes achieved or not – use questions below and evaluation guides: What happened? What was supposed to happen? If there is a difference, why? What is the impact of that difference? What should be learned from this? What improvements might be recommended?
Data Analysis
Conduct Hotwash
Develop timeline of significant events
Analyze performance: Individuals Teams/Functions Outcomes
Hotwash
Player Hotwash: Usually held immediately following exercise
play Typically facilitated by the evaluator
Provides opportunity for: Player self-assessment An interactive discussion Clarification of observed events Assessment of exercise simulations
Timeline Development
Identify the appropriate outcome for each activity
Time Observations Location Team Outcome
0852 Three staff members arrive at JIC – PIO, Deputy, Admin. Ass’t; begin setting up (computers, removing displays from storage, job aids at work stations, etc.)
JIC Rushmore Co.
EM
0905 First media call to JIC requesting info. on event. PIO provides initial incident info. & tells reporter to watch for news release shortly re: JIC activation.
JIC Rushmore Co. PIO
EM
0906 Forest Co. PIO and Assistant arrive at EOC; PIO immediately calls in to EOC
JIC Forest Co. EM
0910 EAS message from EOC received by Fax
JIC EOC EM
0912 EAS copied & distributed at JIC to all staff work stations; additional copies on tables in media room.
JIC JIC staff EM
Make a team timeline of actions
Focus on significantactions
Analysis of Performance
Analysis of activities What tasks were to be accomplished How well were they performed Root causes of differences between
expected and actual performance Recommendations
Root Cause Analysis
1. Why did it happen?
2. Why did that happen?
3. Why was that?
4. And why was that?
5. And why was that?
***Root Cause***
6. And so on…
Each step must completelyexplain the step above…
…down to the basicunderlying causal factor.
Integrated Analysis
Allows further identification of: Successes and best practices New gaps and problems Root causes Recommendations for improvement
Compares observations from different locations and functions
Recommendations for Improvement
Questions for identifying recommendations for improvement: What training and/or equipment is needed? What changes need to be made to plans and
procedures, or organization structures? What changes could be made to the management
processes?
The After-Action Report (AAR)
Serves as feedback tool Summarizes what happened Identifies successes and
recommendations for improvement May include lessons learned to share
with other jurisdictions Help jurisdictions focus resources on
greatest needs
After-Action Report
Prepared in two stages:
Draft AAR – completed immediately after the exercise for review
Community adds improvement steps/corrective actions
Final AAR
AAR Format
Executive Summary Part 1: Exercise Overview Part 2: Exercise Goals and Objectives Part 3: Exercise Events Synopsis Part 4: Analysis of Mission Outcomes Part 5: Analysis of Critical Task
Performance Part 6: Conclusion Appendix A: Improvement Plan Matrix
Improvement Process
Improving preparedness activities: Conduct exercise debrief Identify improvements Finalize AAR Track implementation
Step 5
Conduct Debriefing
Step 5
Conduct Debriefing
Step 6
Identify Improvements
Step 6
Identify Improvements
Step 7
Finalize After Action Report
Step 7
Finalize After Action Report
Step 8
Track Implementation
Step 8
Track Implementation
Exercise Debrief
Provides a forum for jurisdiction officials to: Hear the results of the
analysis Validate the findings and
recommendations in draft AAR
Begin development of Improvement Plan
Improvement Plan
Developed by local jurisdiction during debrief
Identifies how recommendations will be addressed: What actions Who is responsible Timeline for completion
Finalize AAR
Improvement Plan is included in final AAR
Final AAR submitted to ODP through State Administrative Agency
Monitor Implementation
ODP Exercise Management System (under development) will provide: Centralized calendar of exercises across
the country Electronic submission of AAR/IPs to the
SAA and ODP Monitoring of Improvement Plan
implementation
Sharing Lessons Learned
Ready-Net – Web-based, secure information network National repository for best practices
and lessons learned Accessible to approved users within
the response community Administered by the Memorial
Institute for the Prevention of Terrorism
Benefits ofHSEEP Approach
Nationwide consistency
More useful after action reports and improvement plans
Ability of jurisdictions to focus resources on greatest needs
ENHANCED PREPAREDNESS
Exercise EvaluationTraining Course
2 ½ days - Exercise Evaluation methodology
6 sessions to train ODP staff and contractors as change agents (225 people)
Training for SAAs Feb-May 2004 ODP Exercise Design Course being
revised to deliver consistent message
Goal for Working Group
Review and modify Exercise Evaluation Guides for Radiological and Biological attacks Are the right tasks identified? Do other tasks need to added? Are the conditions and typical steps logical
and complete? Are the followup analysis questions the
right questions to assess performance?