HSEEP ExerciseEvaluation and Improvement

ODP’s Mission

Primary responsibility within the executive branch to build and sustain the preparedness of the US to reduce vulnerabilities, prevent, respond to, and recover from acts of terrorism (Homeland Security Act).

ODP’s Responsibilities

Grant programs for planning, equipment, training and exercises

National training program

National exercise program

Grant Programs

State Homeland Security Program

Law Enforcement Terrorism Prevention

Program

Citizen Corps Program

Urban Areas Security Initiative Program

Fire Fighter Assistance Program

State Homeland Security Program

Purpose: to enhance capacity of states and locals to prevent, respond to, and recover from terrorism

Provides funds for Homeland security and emergency operations

planning The purchase of specialized equipment CBRNE and cyber security training programs CBRNE and cyber security exercises State Homeland Security Assessments and

Strategies

Law Enforcement Terrorism Prevention Program

Provide law enforcement communities with funds to support the following prevention activities:

Information sharing to preempt terrorist attacks Target hardening Recognition of potential or developing threats Interoperable communications Intervention of terrorists before they can execute a

threat Planning, organization, training, exercises, and

equipment

Citizen Corps Program

Provides funds to support Citizen Corps Councils with planning, outreach, and management of Citizen Corps program and activities

Form and sustain a Citizen Corp Council Engage citizens in homeland security Conduct public education and outreach Develop and implement Citizen Corps programs Coordinate Citizen Corps activities with other DHS

funded programs and other federal initiatives

Urban Areas Security Initiative Program

Address the unique needs of large urban areas – 50 cities

Conduct jurisdictional assessment and develop Urban Area Homeland Security Strategy.

Funds for planning, equipment, training, exercise, and administration and operational activities related to heightened threat levels

Fire Fighter Assistance

Protect public and fire fighters against fire and fire-related hazards Fire fighting Operations and Safety Fire Prevention Fire fighting Vehicles

Strategy Process Overview

END RESULTEND RESULT = Capability Improvements

Statewide HomelandStatewide HomelandSecurity StrategySecurity Strategy

Conducted at the local and state levels

Created at the regional and state level

Created by ODP

AssessmentsAssessments State AssistanceState AssistancePlanPlan

State and Urban Area use strategy to identify & allocate all HS resources

Public Health

Health Care

Public Works

Government Administrative

Private Sector

Non-Profit/Voluntary Sector

State and local jurisdictions All First Responder Disciplines

Strategy Participants

Fire Service

HazMat

Emergency Medical Services

Law Enforcement

Emergency Management

Public Safety

Communications

Assessment Overview

Shortfalls or “Gaps”

Agricultural Vulnerability Assessment

* CBRNE: Chemical, Biological, Radiological, Nuclear, Explosive

ThreatAssessment

VulnerabilityAssessment

Planning Factors

CBRNE* Scenarios

RequiredCapabilities

CurrentCapabilities

Risk Assessment Needs Assessment

StatewideHomelandSecurityStrategy

Threat Assessment

Agricultural Vulnerability Assessment

Risk Assessment

ThreatAssessment

VulnerabilityAssessment

Who:

Local, state, and federal law enforcement officials

What: • Identify number of Potential

Threat Elements (PTEs)

• Identify threat factors (existence, violent history, intentions, WMD capability, and targeting)

• Identify motivations (political, religious, environmental, racial, or special interest)

• Identify WMD capabilities (CBRNE)

Vulnerability Assessment

Risk Assessment

ThreatAssessment

Who:

All response disciplines at local, state, and federal levels

What:Identify critical infrastructure/ potential targets

Evaluate targets for:

• Level of visibility

• Criticality of target site

• Impact outside of jurisdiction

• Access to target

• Target threat of hazard

• Target site population capacity

• Potential collateral mass casualties

Agricultural Vulnerability Assessment

VulnerabilityAssessment

The results from the risk assessment process (threat and vulnerability) provide a link to the capabilities and needs assessment process.PlanningOrganizationEquipmentTrainingExercises

Capabilities and Needs: Planning

State Homeland SecurityStrategy

Developed by State based on local needs

Provides blueprint for planning of homeland security efforts to enhance preparedness and for use of resources

StateAssistance Plans

ODP uses the strategies and needs assessment data to tailor and formulate a State/Metro Assistance Plan (SAP/MAP) for each state

A SAP/MAP is a blueprint for the delivery of ODP training, exercise, technical assistance and equipment services

National Training Program

Training for federal, state and local homeland security professionals

Based on critical tasks to prevent, respond to or recover from a terrorist incident

Over 40 courses available

ODP Training Program

ODP offers more than 40 courses (Examples)

Live chemical agents training – Center for Domestic Preparedness

Live explosives training – New Mexico Institute of Mining and Technology

Radiological and nuclear agents training – Nevada Test Site

Advanced emergency medical training using human patient simulators – Texas A&M

Training on bioterrorism – Louisiana State University

National Exercise Program

Responsible for National Exercise Program

Threat and performance-based excises at federal, state, local, and international levels

Strategy and Exercise Planning Workshops to define exercise needs and plan for each state

Assess Program Success Through Exercises

Performance measures for ODP’s grant, training, and exercise programs are tied to performance of critical tasks

Percent of jurisdictions that can perform critical tasks as demonstrated through exercises 500,000+ population 100,000+ population 50,000+ population

Overview of HSEEP

Threat- and Performance-based Exercises

Cycle of exercises

Increasing complexity

To improve preparedness

HSEEP Manuals

Volume I: Program Overview and Doctrine

Volume II: Exercise Evaluation and Improvement

Volume III: Exercise Development Volume IV: Sample Exercise

Documents and Formats

Vol I: HSEEP Overview and Doctrine

ODP’s exercise and evaluation doctrine Uniform approach for exercise design,

development, conduct, and evaluation Exercise design and implementation

process Suite of common scenarios (TBD)

Vol II: Exercise Evaluation and Improvement

Defines exercise evaluation and improvement process

Provides uniform set of evaluation guides

Defines data analysis process Includes standardized After-Action

Report template

Vol III: Exercise Development

Defines exercise planning and design process

Provides guidance for the development and conduct of various types of exercises

Vol IV: Sample Documents

Provides sample letters, planning documents, checklists, scenarios, etc.

Reduces development time for exercise design team

Exercise Evaluation

Assess preparedness at federal, state and local levels

Validate strengths and identify improvement opportunities, resulting in improved preparedness

Provide guide for resource allocations

Evaluation Enhancements

Focus on performance of critical tasks and mission outcomes

Use of uniform evaluation tools Enhanced data analysis Debriefing meeting with key officials Improvement Plan Track implementation of improvements Suite of common scenarios (TBD)

Exercise Evaluation Methodology Development

Exercise Evaluation Working Group

Builds on Responder Guidelines

ODP exercise experience

CSEP and other programs

Will continue to enhance and improve

Exercise Evaluation and Improvement Process

Exercise Evaluation and Improvement Process Data Collection and Analysis

Step 1

Plan & Organize the Evaluation

Step 1

Plan & Organize the Evaluation

Step 2

Observe the Exercise &

Collect Data

Step 2

Observe the Exercise &

Collect Data

Step 3

Analyze Data

Step 3

Analyze Data

Step 4

Develop After Action Report

Step 4

Develop After Action Report

Improving Preparedness

Step 5

Conduct Debriefing

Step 5

Conduct Debriefing

Step 6

Identify Improvements

Step 6

Identify Improvements

Step 7

Finalize After Action Report

Step 7

Finalize After Action Report

Step 8

 Track Implementation

Step 8

 Track Implementation

Evaluation Planning, Observation, and Analysis

Levels of Analysis

Performance is analyzed at three levels: Task level Agency/discipline/function level Mission level (within and across

communities)

Levels of Analysis

Task Level Performance Answers the question: did

the person or team do the right thing the right way at the right time?

Helps assess need for training, equipment, personnel, etc.

Task = work with measurable output that has utility

Levels of Analysis

Agency/Discipline/Function Level Performance — Multiple teams Answers the question: did the larger team or

organization perform duties in accordance with plans and policies?

Helps assess communication, coordination, planning budgets, etc.

Levels of Analysis

Mission Level Performance Answers the question: were the

mission level outcomes achieved?

Addresses jurisdictional preparedness

Outcomes = results

Mission Outcomes

Pre-Event

Emergency Response

Post-Event

Prevention/Deterrence

Emergency Assessment Emergency Management Hazard Mitigation Public Protection Victim Care Investigation/Apprehension

Recovery/Remediation

Evaluation Requirements

Determine what outcomes will be evaluated, based on exercise objectives

Identify activities to be evaluated Identify which functions should be

observed Determine where observations will take

place Identify the appropriate evaluation tools

Exercise Evaluation Guides

ODP has developed Exercise Evaluation Guides that: Identify the activities that the evaluator

should be observing Provide consistency in tasks across

exercises Link individual tasks to disciplines and

outcomes

The EVALPLAN

Exercise-specific information

Plans, policies, procedures, and agreements

Evaluator recruiting and assignments

Evaluator training and instructions

Recruiting and Assigning Evaluators

Setting expectations – evaluators must be available for: pre-exercise training and briefing pre-exercise site visit the entire exercise (hours to days) post-exercise hot-wash post-exercise data analysis (1 day) contribution to the draft AAR

Recording Observations

The emphasis is on Who? What? When? Where? How? Why?

Record observations through: use of Evaluator Guides blank sheets of paper

Collect exercise documents

Record Significant Activities

Initiating scenario events Facility activities Response actions Key decisions made by Players Deviations from plans and procedures Completion time of events

Evaluator Summary

Compile observations into chronological narrative of events

Describe outcomes achieved or not – use questions below and evaluation guides: What happened? What was supposed to happen? If there is a difference, why? What is the impact of that difference? What should be learned from this? What improvements might be recommended?

Data Analysis

Conduct Hotwash

Develop timeline of significant events

Analyze performance: Individuals Teams/Functions Outcomes

Hotwash

Player Hotwash: Usually held immediately following exercise

play Typically facilitated by the evaluator

Provides opportunity for: Player self-assessment An interactive discussion Clarification of observed events Assessment of exercise simulations

Timeline Development

Identify the appropriate outcome for each activity

Time Observations Location Team Outcome

0852 Three staff members arrive at JIC – PIO, Deputy, Admin. Ass’t; begin setting up (computers, removing displays from storage, job aids at work stations, etc.)

JIC Rushmore Co.

EM

0905 First media call to JIC requesting info. on event. PIO provides initial incident info. & tells reporter to watch for news release shortly re: JIC activation.

JIC Rushmore Co. PIO

EM

0906 Forest Co. PIO and Assistant arrive at EOC; PIO immediately calls in to EOC

JIC Forest Co. EM

0910 EAS message from EOC received by Fax

JIC EOC EM

0912 EAS copied & distributed at JIC to all staff work stations; additional copies on tables in media room.

JIC JIC staff EM

Make a team timeline of actions

Focus on significantactions

Analysis of Performance

Analysis of activities What tasks were to be accomplished How well were they performed Root causes of differences between

expected and actual performance Recommendations

Root Cause Analysis

1. Why did it happen?

2. Why did that happen?

3. Why was that?

4. And why was that?

5. And why was that?

***Root Cause***

6. And so on…

Each step must completelyexplain the step above…

…down to the basicunderlying causal factor.

Integrated Analysis

Allows further identification of: Successes and best practices New gaps and problems Root causes Recommendations for improvement

Compares observations from different locations and functions

Recommendations for Improvement

Questions for identifying recommendations for improvement: What training and/or equipment is needed? What changes need to be made to plans and

procedures, or organization structures? What changes could be made to the management

processes?

The After-Action Report (AAR)

Serves as feedback tool Summarizes what happened Identifies successes and

recommendations for improvement May include lessons learned to share

with other jurisdictions Help jurisdictions focus resources on

greatest needs

After-Action Report

Prepared in two stages:

Draft AAR – completed immediately after the exercise for review

Community adds improvement steps/corrective actions

Final AAR

AAR Format

Executive Summary Part 1: Exercise Overview Part 2: Exercise Goals and Objectives Part 3: Exercise Events Synopsis Part 4: Analysis of Mission Outcomes Part 5: Analysis of Critical Task

Performance Part 6: Conclusion Appendix A: Improvement Plan Matrix

Improvement Process

Improving preparedness activities: Conduct exercise debrief Identify improvements Finalize AAR Track implementation

Step 5

Conduct Debriefing

Step 5

Conduct Debriefing

Step 6

Identify Improvements

Step 6

Identify Improvements

Step 7

Finalize After Action Report

Step 7

Finalize After Action Report

Step 8

Track Implementation

Step 8

Track Implementation

Exercise Debrief

Provides a forum for jurisdiction officials to: Hear the results of the

analysis Validate the findings and

recommendations in draft AAR

Begin development of Improvement Plan

Improvement Plan

Developed by local jurisdiction during debrief

Identifies how recommendations will be addressed: What actions Who is responsible Timeline for completion

Finalize AAR

Improvement Plan is included in final AAR

Final AAR submitted to ODP through State Administrative Agency

Monitor Implementation

ODP Exercise Management System (under development) will provide: Centralized calendar of exercises across

the country Electronic submission of AAR/IPs to the

SAA and ODP Monitoring of Improvement Plan

implementation

Sharing Lessons Learned

Ready-Net – Web-based, secure information network National repository for best practices

and lessons learned Accessible to approved users within

the response community Administered by the Memorial

Institute for the Prevention of Terrorism

Benefits ofHSEEP Approach

Nationwide consistency

More useful after action reports and improvement plans

Ability of jurisdictions to focus resources on greatest needs

ENHANCED PREPAREDNESS

Exercise EvaluationTraining Course

2 ½ days - Exercise Evaluation methodology

6 sessions to train ODP staff and contractors as change agents (225 people)

Training for SAAs Feb-May 2004 ODP Exercise Design Course being

revised to deliver consistent message

Goal for Working Group

Review and modify Exercise Evaluation Guides for Radiological and Biological attacks Are the right tasks identified? Do other tasks need to added? Are the conditions and typical steps logical

and complete? Are the followup analysis questions the

right questions to assess performance?