OFFICE OF SECURITY ASSESSMENTS APPRAISAL PROCESS …

OFFICE OF SECURITY ASSESSMENTS APPRAISAL PROCESS PROTOCOLS

January 2017

Office of Security Assessments Office of Cyber and Security Assessments

Office of Enterprise Assessments U.S. Department of Energy

Office of Security Assessments Appraisal Process Protocols

January 2017

OFFICE OF SECURITY ASSESSMENTS APPRAISAL PROCESS PROTOCOLS

Approved by: ______________________________________ Date: ________________

James Lund, Acting Director

Office of Security Assessments

Office of Cyber and Security Assessments

1/31/2017

Office of Security Assessments Appraisal Process Protocols Preface

January 2017 i

Preface

The U.S. Department of Energy (DOE) Office of Security Assessments (EA-22), within the Office of

Enterprise Assessments (EA) Office of Cyber and Security Assessments, is responsible for conducting

independent oversight appraisals of safeguards and security (S&S) programs at DOE sites and other

organizations. These assessments focus on high consequence activities, such as high hazard nuclear

operations, and the protection of high value security assets, such as Category I quantities of special nuclear

material and classified information assets, as mandated by DOE Orders 227.1A, Independent Oversight

Program, and 226.1B, Implementation of Department of Energy Oversight Policy.

The EA-22 Appraisal Process Protocols, in conjunction with the Independent Oversight Program Appraisal

Process Protocols issued by EA in December 2015, describes the principal activities for implementing the

DOE independent oversight program. This document also describes the processes, techniques, and

procedures that EA-22 uses to evaluate DOE (including National Nuclear Security Administration) and

contractor organizations’ programs (referred to as “sites” throughout the document) designed to protect

national security assets, including special nuclear material and sensitive and classified information. EA-22

accomplishes these evaluations through rigorous and objective appraisals of the effectiveness of DOE

security policies and the performance of DOE line management in implementing those policies. EA-22

assessments are also key in identifying emerging trends throughout DOE.

Major additions in this revision of EA-22 Appraisal Process Protocols include referencing the limited-notice

testing program, processes for sharing lessons learned from appraisal activities, and greater focus on the

insider threat. Limited-notice performance testing focuses on specific S&S elements, in contrast to multi-

topic assessments that provide a more complete evaluation of overall performance. A goal of this approach

is to gather data with minimal advance notice to the personnel and facilities being tested, in order to

provide a realistic appraisal under actual operating conditions. EA-22 has also initiated mechanisms for

sharing lessons learned with line security managers and the National Training Center to enhance local

programs and S&S training for all of DOE. EA-22’s approach for evaluating the insider threat uses

enhanced lines of inquiry to determine the effectiveness of site administrative and physical security

measures.

This protocol document is part of an ongoing effort to ensure quality and consistency in DOE independent

oversight activities while providing the flexibility to address all activities being evaluated. To ensure that

this protocol remains current, all users of this protocol are encouraged to provide comments and

recommendations to EA-22 for consideration.

Office of Security Assessments Table of Contents Appraisal Process Protocols

ii January 2017

Table of Contents Acronyms ..................................................................................................................................................... iii

Definitions ................................................................................................................................................... iv

Section 1. Introduction ................................................................................................................................. 1

Mission ............................................................................................................................................ 1

Roles and Responsibilities ............................................................................................................... 1

Appraisal Types ............................................................................................................................... 3

Augmentation and Observer Program ............................................................................................. 4

Section 2. Appraisal Process ........................................................................................................................ 6

2.1 Appraisal Planning .................................................................................................................... 6

Appraisal Scheduling Process .......................................................................................................... 6

Appraisal Scoping ............................................................................................................................ 6

Announcement Memorandum ......................................................................................................... 7

Logistics Planning............................................................................................................................ 8

Site Scoping Visit ............................................................................................................................ 8

Performance Test Scoping Visit ...................................................................................................... 8

Site Cooperation and Coordination .................................................................................................. 8

Assessment Plan Development ........................................................................................................ 9

Planning Meetings ........................................................................................................................... 9

2.2 Appraisal Conduct .................................................................................................................... 9

Data Collection ................................................................................................................................ 9

Validation ........................................................................................................................................ 9

Data Analysis ................................................................................................................................. 10

End-of-Day Team Meetings and Morning Manager’s Meetings ................................................... 10

Report Writing ............................................................................................................................... 10

Exit Meeting .................................................................................................................................. 11

2.3 Post-Appraisal Activities ........................................................................................................ 12

Post-Appraisal Validation Review ................................................................................................. 12

Comment Resolution and Technical Edit ...................................................................................... 12

Final Report ................................................................................................................................... 12

Briefings ........................................................................................................................................ 12

Sharing Lessons Learned ............................................................................................................... 13

Appendix A: Multi-Topic Assessment Timeline ..................................................................................... A-1

Appendix B: Example Document Request .............................................................................................. B-1

Appendix C: Example of Multi-Topic Assessment Plan ......................................................................... C-1

Appendix D: Appraisal Scheduling Process ............................................................................................ D-1

Appendix E: Factors for Determining a Site’s Appraisal Priority, Type, and Scope .............................. E-1

Appendix F: Data Analysis, Deficiencies, Findings, Opportunities for Improvement,

Recommendations, Best Practices, and Ratings................................................................. F-1

Office of Security Assessments Appraisal Process Protocols Acronyms

January 2017 iii

Acronyms

CAS Central Alarm Station

CFR Code of Federal Regulations

CMPC Classified Matter Protection and Control

COMPASS Comprehensive Analyses of Safeguards Strategies

DOE U.S. Department of Energy

EA Office of Enterprise Assessments

EA-22 Office of Security Assessments

FCL Facility Clearance

FOCI Foreign Ownership, Control or Influence

FOF Force-on-Force

FY Fiscal Year

HRP Human Reliability Program

IOSC Incidents of Security Concern

LNPT Limited-Notice Performance Test

LSPT Limited-Scope Performance Test

MAA Material Access Area

MBA Material Balance Area

MC&A Material Control and Accountability

MOA Memorandum of Agreement

MOU Memorandum of Understanding

NNSA National Nuclear Security Administration

NTC National Training Center

OPSEC Operations Security

PAP Performance Assurance Program

POC Point of Contact

PPM Protection Program Management

PSS Physical Security Systems

QRB Quality Review Board

S&S Safeguards and Security

SAS Secondary Alarm Station

SCI Sensitive Compartmented Information

SECON Security Conditions

SNM Special Nuclear Material

SPO Security Police Officer

SSP Site Security Plan

TAP Training Approval Program

TID Tamper-Indicating Device

TSCM Technical Surveillance Countermeasures

UFV&A Unclassified Foreign Visits and Assignments

VA Vulnerability Assessment

VAR Vulnerability Assessment Report

VTR Vault-Type Room

Office of Security Assessments Definitions Appraisal Process Protocols

iv January 2017

Definitions

Deficiency: An inadequacy in the implementation of an applicable requirement or performance standard

identified during an appraisal. Deficiencies may serve as the basis for one or more findings. [DOE Order

227.1A]

DOE Oversight: DOE oversight encompasses activities performed by DOE organizations to determine

whether Federal and contractor programs and management systems, including assurance and oversight

systems, are performing effectively and complying with DOE requirements. Oversight programs include

operational awareness activities, onsite reviews, assessments, self-assessments, performance evaluations,

and other activities that involve evaluation of Federal and contractor organizations that manage or operate

DOE sites, facilities, or operations.

Evaluators: Individuals assigned the responsibility of formally assessing the readiness and performance

effectiveness of safeguards and security (S&S) system elements during the conduct of a performance test

as part of an independent oversight appraisal activity. Evaluators must be trained to perform their assigned

duties.

Limited-Notice Performance Test (LNPT): A performance test that is coordinated and scheduled with

one or more site trusted agents but is conducted without prior announcement to other site personnel.

Limited-notice tests are intended to elicit the most accurate information regarding an individual’s

knowledge or the performance of S&S programs.

Limited-Scope Performance Test (LSPT): A performance test designed to validate or evaluate specific

skills, equipment, or procedures instead of testing an entire system or multiple subsystems. An LSPT may

be interrupted to facilitate data collection, and the LSPT may be revised or restarted in order to achieve

specific goals.

Trusted Agents: Individuals who have appropriate operational authority or a compartmented role

necessary to provide administrative and logistical support for coordinating and conducting independent

oversight and LNPT activities. Trusted agents are responsible for maintaining strict confidentiality of

performance testing information in the interest of test validity. Trusted agents must remain impartial in

validating and developing performance test parameters and events necessary to evaluate identified

objectives. The utmost diligence must be applied to limit the number of trusted agents to the minimum

needed to administratively and logistically conduct the test.

Office of Security Assessments Appraisal Process Protocols Introduction

January 2017 1

Section 1 – Introduction

Mission

The mission of the U.S. Department of Energy (DOE) Office of Security Assessments (EA-22), within the

Office of Enterprise Assessments (EA), is to provide feedback to internal and external stakeholders by

conducting independent evaluations of the effectiveness of safeguards and security (S&S) policies and

programs throughout DOE. The major intention of these assessments is to foster continuous improvement

in the Department’s security program by identifying deficiencies, positive attributes, and unacknowledged

program risks and communicating them effectively to DOE management. To perform this mission, EA-22

plans and conducts a variety of announced and unannounced appraisal activities and performance testing

that incorporate a broad range of threats and scenarios in order to provide a complete and realistic evaluation

of site S&S system’s readiness to protect DOE assets, with an emphasis on protection of Category I special

nuclear material (SNM) and highly sensitive or classified information. EA-22 develops and validates

appraisal results in reports that may identify best practices, findings, deficiencies, and opportunities for

improvement. EA-22 also performs follow-up assessments to ensure that site-specific corrective actions

are effective and conducts complex-wide reviews of S&S programs to enhance the overall effectiveness

and efficiency of these programs.

Roles and Responsibilities

EA-22 Director (or Deputy Director)

In addition to the roles and responsibilities set forth in the EA Independent Oversight Program Appraisal

Process Protocols, the EA-22 Director:

Provides overall direction and manages the S&S independent oversight program.

Directs self-assessment program/activities.

Ensures that upcoming assessment activities are coordinated with line management and

stakeholders in a collaborative manner, well in advance of execution.

Briefs senior stakeholders, including the Under Secretaries, Secretarial Officers, the EA Director,

and DOE policy organizations, on the results of appraisal activities.

Ensures that appraisal feedback sessions are conducted with line management as appropriate.

Notifies senior EA management and the DOE Office of the Inspector General when appraisal

activities identify concerns that may have criminal or waste/fraud/abuse implications.

Develops and maintains assessment guides for conducting S&S appraisals.

Works with cognizant DOE line managers to resolve disagreements on appraisal schedules, results,

findings, or ratings, and works with cognizant policy organizations to ensure proper application

and characterization of DOE policies and directives in appraisal reports.

Ensures that the quality of independent oversight activities and reports is maintained.

Serves as appraisal leader for assessments as needed or required.

Coordinates appraisal team structure, designates appraisal Topic Team Leaders, and works with

stakeholders to define assessment scope.

Serves on the Quality Review Board (QRB).

Office of Security Assessments Introduction Appraisal Process Protocols

2 January 2017

Team Chief or Deputy Team Chief

Plans and leads assessments of information/physical security programs or topics as directed.

Provides input on recommended assessment scope.

Provides direction and guidance to team members on the approach to specific data collection

activities.

Oversees the development of assessment plans.

Provides feedback on proposed assessment team structure and makes recommendations for allocation of resources needed to accomplish the scope.

Makes arrangements with the site for document requests and other logistics as needed.

Communicates upcoming assessment activities to appropriate field and program office

management well in advance of the scheduled activity.

Seeks feedback on special interest items, implementation, and scope.

Establishes the schedule of events for security assessments and makes specific assignments.

Ensures that team members perform assigned data collection and performance testing duties in

accordance with DOE Order 227.1A, Independent Oversight Program, relevant DOE policy, and

EA protocols.

Addresses site concerns associated with data collection activities.

Provides daily feedback to site personnel to validate assessment information and clearly

communicate areas of concern.

Oversees the development of draft assessment reports.

Briefs site Federal and contractor management and security personnel on assessment results and

provides a written synopsis (letter form) of assessment results to site management prior to departure

if a draft report has not been developed.

If a significant vulnerability is identified, notifies EA management and the site Federal S&S

Director in accordance with DOE Order 227.1A, Independent Oversight Program.

Topic Team Leaders

Support Team Chief/Deputy in the assessment of physical/information security programs.

Lead specific topic teams during appraisal activities as needed or required.

Provide direction and guidance to topic team members on the approach to physical/information

security data collection activities, including performance testing.

Update and maintain assessment guide(s) for assigned topic(s).

Maintain current knowledge of applicable assigned S&S topic requirements and conduct training

with team members as required.

Provide input to the Team Chief/Deputy on topic area document requests (see Appendix B for a

baseline document request list), lines of inquiry (see Appendix C, Example of Multi-Topic

Assessment Plan, for generally used lines of inquiry), and other necessary logistics to support the

team.

Provide feedback on proposed topic team structure and make recommendations for allocation of resources needed to accomplish the scope.

Establish a topic team assessment schedule and make specific data collection assignments.

Ensure that team members perform their assigned duties.

Address site concerns associated with data collection or performance testing activities.

Provide daily feedback to site personnel to validate assessment information and clearly

communicate areas of concern.

Participate in briefing site management and security personnel on data collection results, as required.

Coordinate the preparation of topic input to the draft assessment report.


January 2017 3

Work with the Team Chief/Deputy to resolve site comments on the appraisal report.

Team Members

Support the Team Chief/Deputy and applicable Topic Team Leader in conducting appraisals of

security programs.

Provide input to the Topic Team Leader on topic scope and potential approaches for accomplishing

security appraisals.

Conduct appraisal process activities following the direction and guidance of Topic Team Leaders.

Assist in preparing the schedule of interviews to accomplish during the onsite visit.

Review key site security documents before the onsite visit.

Conduct thorough and fair evaluations and assessments in accordance with the assessment plan.

Validate assessment data and conclusions with site personnel on a daily basis to ensure factual

accuracy.

Participate in briefing site management and information/physical security personnel on assessment

results, if requested.

Provide written input for topical sections of draft reports as directed by the Topic Team Leader.

Work with the Topic Team Leader to resolve site comments on the draft report.

Appraisal Types

EA-22 conducts a broad array of appraisals, including:

Multi-Topic Assessments: Multi-topic assessments evaluate the security functional areas related

to the protection of SNM or classified and sensitive unclassified information, and site actions for

deterring, detecting, and mitigating the insider threat. Multi-topic assessments typically require the

largest amount of resources, as well as time on site with two weeks of data collection activities and

another week for developing a draft report. A key aspect of EA-22 assessments is the conduct of

a wide range of performance tests, including large-scale force-on-force (FOF) performance tests

consistent with the Department’s threat policy (i.e., DOE Order 470.3B, Graded Security

Protection Policy, and DOE Order 470.3C, Design Basis Threat Policy, as applicable) and limited-

notice performance tests (LNPTs) and/or limited-scope performance tests (LSPTs). See Appendix

C for an example of a Multi-Topic Assessment Plan for more details.

Targeted Assessments: Targeted assessments are conducted to address S&S concerns that

transcend performance at a specific site or location. They might address the effectiveness of

program elements as implemented across DOE by analyzing DOE complex-wide program issues,

or they might analyze the implementation of a specific policy item throughout the complex.

Targeted assessments are also performed to address an area, concern, deficiency, or weakness

within a program, focusing (for example) on the status of a specific program element, the adequacy

of specific policies, or the implementation status of specific policies throughout DOE.

Follow-up Assessments: EA conducts follow-up assessments to determine the status and progress

of corrective actions and other actions taken in response to deficiencies identified during previous

EA appraisals or DOE line management oversight activities, focusing on evaluating the

effectiveness and sustainability of corrective actions. Follow-up assessments can include

announced and limited-notice performance testing and FOF performance tests. The scope and team

size for these assessments can vary, and data collection activities are tailored to assess the overall

effectiveness of corrective actions.

Office of Security Assessments Introduction Appraisal Process Protocols

4 January 2017

Limited-Notice Performance Testing: EA-22 uses limited-notice testing and conducts

evaluations outside of pre-scheduled appraisal activities to enhance DOE senior leadership’s and

line managers’ understanding of protection program readiness. LNPT activities related to physical

and information security are necessarily narrow in scope and limited in team size. Accordingly,

EA-22 maintains a set of physical and information security LNPT objectives and associated tasks,

conditions, standards, and evaluation criteria. The EA Limited-Notice Performance Test Guide

provides specific guidance for physical security LNPTs.

Protective Force Assessments: These assessments focus primarily on the implementation of the

DOE requirements for armed protective forces at Departmental facilities that do not possess SNM.

These reviews may also include specific aspects of other topical areas, particularly protection

program management and physical security systems, as deemed necessary. Data collection

activities are tailored to the assessment’s scope, including document reviews, interviews, and

LSPTs. The assessment teams generally consist of a small number of subject matter experts who

conduct onsite activities over the course of several days. Assessment planning, execution, and

post-appraisal activities are generally conducted in accordance with the processes described in

Section 2.

Augmentation and Observer Program

EA-22 recognizes the importance of working closely with Headquarters program offices, field offices, and

DOE and National Nuclear Security Administration (NNSA) physical and information security

organizations to ensure that appraisal results are clearly communicated and identified deficiencies are

adequately addressed. In addition to pre-coordination, EA has implemented an augmentation and observer

program that includes DOE Federal or contractor subject matter experts as augmentees or observers on

assessment and review teams.

The augmentee program allows the participants to observe the inner workings of the appraisal process and

return to their home organizations with information on oversight and protection program best practices.

The augmentee is considered an assessor and member of the EA-22 appraisal team. The field augmentee

program has proven to be mutually beneficial, developing a high level of trust between EA-22 and

participating sites and fostering improved relations during future appraisals.

The observer program offers benefits similar to the augmentee program; however, the observer is not

involved in data collection activities and is not considered an assessor.

Requesting organizations must follow these general program concepts to ensure the integrity of EA-22’s

appraisal process:

The DOE/NNSA augmentee is recommended in writing (emails are acceptable) by the applicable

Headquarters or field office and is selected and approved for participation by the EA-22 Office

Director or Deputy Director. Recommendations must come from the senior Federal S&S manager

and must include the specific objective and overall intention of the augmentee’s participation.

o Augmentees cannot participate in appraisals at their own site or organization; contractor

augmentees are further restricted from participating in appraisals at other sites operated by

their employer.


January 2017 5

o Augmentees are fully integrated into the appraisal team and participate in the data

collection activities of the topic team to which they are assigned.

DOE and other government agency observers are recommended in writing (emails are acceptable)

by the applicable Federal agency manager and then are approved by EA-22 Office Director or

Deputy Director. Recommendations must come from the senior Federal S&S manager and must

include the specific objective and overall intention of the observer’s participation.

Observers are assigned to one or more topic teams during an appraisal activity but do not conduct

data collection activities.

Office of Security Assessments Appraisal Process Appraisal Process Protocols

6 January 2017

Section 2 – Appraisal Process

Each EA-22 assessment is a continuous process involving a myriad of activities at the strategic, program,

and individual appraisal level. Appraisal activities focus on identifying and executing activities that

collectively address the most significant security vulnerabilities faced by the Department, in order to

provide recommended actions to the Secretary and other senior managers for their use in allocating

resources and targeting remedial actions in a timely manner.

The appraisal process consists of pre-appraisal activities (planning, including scoping visits), execution

(conducting and documenting the assessment), and post-appraisal activities (closeout). This section of the

protocol describes these activities for multi-topic assessments in a normal sequence and an approximate

timeline within which they are completed. It should be noted that the other appraisal types (targeted, follow-

up, and protective force assessments and LNPTs) generally mirror this process. For some assessment

activities, EA management may determine that ratings will be assigned. In such cases, the appraisal team

is responsible for recommending ratings, and the EA Director approves the final ratings, with input from

the director of EA-22 (see EA Independent Oversight Program Appraisal Process Protocols for additional

information).

From the start of the scoping to the closeout and final publication of the appraisal report, the typical multi-

topic assessment is a 30-week process. The actual amount of onsite planning and data collection varies

with the type of appraisal being conducted; however, for multi-topic assessments, major onsite assessment

activities occur in two visits constituting three weeks (supplemented with short-duration scoping and

planning visits by a small number of planners and managers to ensure efficient and effective planning of

major assessment activities). The appraisal planning, conduct, and follow-on activities are described in

Sections 2.1, 2.2, and 2.3, respectively. A detailed timeline diagram indicating the sequence and timing of

these activities for a typical multi-topic assessment is provided in Appendix A. Appendices B through F

provide supplemental detail for selected phases of the process.

2.1 Appraisal Planning

Appraisal Scheduling Process

EA-22 develops its annual schedule of appraisals by using the overall process described in Appendix D of

this protocol. Throughout this process, EA-22 engages Headquarters program office and field management

and security officials. Although this process results in a formal schedule for the year, ongoing interaction

between EA-22 and DOE/NNSA line managers often results in modifications to either the scope or the

schedule of the appraisals. While these modifications can be challenging for EA-22 managers and staff,

this flexibility allows for optimization of appraisal scope and timing and, ultimately, the appraisal’s value

to line managers.

Appraisal Scoping

EA-22 determines the appraisal type best suited to accomplish line management and EA’s objectives, as

well as the scope of the appraisal activity (topics evaluated, level of detail, and lines of inquiry), by

coordinating with senior DOE/NNSA leadership and line management. EA-22 uses the data gathered

during the scoping process to begin developing a detailed appraisal plan. The list below shows some of the

factors that EA-22 considers when making these determinations (see Appendix E, Table E-1, Factors for

Determining a Site’s Appraisal Priority, Type, and Scope, for additional information on how these factors

relate to a site’s priority for evaluation). This risk-based assessment scoping methodology provides a

Office of Security Assessments Appraisal Process Protocols Appraisal Process

January 2017 7

mechanism for determining the emphasis to be placed on particular requirements and assets during EA-22

appraisal planning, data collection, analysis, and reporting.

Asset characterization and site mission: The types and attractiveness of security interests located

at a DOE site and the associated risks (or perceived risks) are significant considerations in

determining the scope and type of appraisal activity to be conducted.

Changes in site mission and operations: Significant changes in site mission/operations, site

geography, or the site security program’s contract structure are factors that EA-22 evaluates in the

appraisal scoping process.

Changes in policy: DOE policy and other applicable national standards are the fundamental

measure by which EA-22 conducts its appraisal activities. Changes in policy can significantly

impact site operations and thus are important considerations in determining the appraisal scope.

Previous evaluation results: Previous EA assessment results and regulatory enforcement actions,

DOE Inspector General inspections, site/operations office surveys, and contractor assurance

activities, as well as evaluation results from external agencies, provide valuable insights that

EA-22 considers during the appraisal scoping process.

Incidents of security concern: Recent/ongoing major vulnerabilities, significant incidents, or

chronic recurrence of lesser incidents influence EA-22’s scoping and prioritization process.

Line management input: EA-22 is committed to supporting Headquarters and field line managers

in accomplishing their S&S responsibilities and achieving DOE mission objectives. Therefore,

input regarding requested focus areas from Headquarters and field line management is an especially

important factor in determining appraisal priority, type, and scope.

Announcement Memorandum

Once EA senior management has approved the annual schedule and EA-22 has determined the scope of an

appraisal activity, an announcement memorandum is issued to the affected site approximately 90 days

before the start of the appraisal. The memorandum provides the dates, scope, and any special considerations

for the appraisal. EA-22 ensures that key program office officials receive a copy of the memorandum.

Separately, the EA-22 appraisal Team Chief communicates a data call request to the designated site point

of contact (POC) to ensure that the site has ample time to compile and provide the requested materials. The

data call request includes a detailed listing of documents, drawings, diagrams, and other materials being

requested to support planning and conduct of the appraisal activity. Appendix B provides examples of data

call document requests for each topical area.

Data call items are typically of two types: (1) the initial materials for EA-22’s review during the planning

process, and (2) other materials that should be available for onsite review. The assessment team reviews

the initial data call items during the Headquarters planning meeting before the team travels to the site. The

results of these initial document reviews may lead one or more teams to request additional data and

documentation from the site. The site sends the initial data call items to the EA-22 office in Germantown,

Maryland. Approximately four weeks before the onsite FOF performance testing begins, EA sends these

materials back to the site.


8 January 2017

Logistics Planning

The EA administrative staff (normally two individuals) is instrumental in planning for and executing

various tasks associated with onsite appraisal activities. Activities include identifying team workspace(s);

coordinating team lodging and issuing the appraisal team lodging announcement memo; arranging with the

site to provide the requisite team access, classified storage containers, copiers, and shredders; coordinating

EA computer support; and communicating any additional support requirements to the site.

Site Scoping Visit

EA-22 contacts site managers several times early in planning and conducts a one-day site scoping visit so

the Team Chief and Deputy, along with members of the administrative staff, can meet face to face with

field DOE/NNSA line management and S&S personnel to discuss the upcoming appraisal, including the

appraisal process and the tentative scope and focus of the appraisal. Topic Team Leaders may also

participate in these visits. The site scoping visit helps clarify the details of data collection and performance

testing activities, the site’s safety and security training requirements, and workspace availability.

Performance Test Scoping Visit

As the start of the onsite appraisal activity approaches, a performance test scoping visit brings together the

EA performance test director and coordinator for large-scale performance FOF tests and supporting LSPTs

and the site’s trusted agents and senior controller. The objectives of the visit are to identify valid and

realistic scenarios/tests and establish performance criteria. These objectives also necessitate interaction and

discussion of targets with site vulnerability assessment staff and local Federal line management officials

and staff.

Site Cooperation and Coordination

The cooperation and assistance of DOE/NNSA site representatives are essential for an effective appraisal.

Local representatives and trusted agents provide detailed knowledge of the site and systems, arrange

administrative and logistical support, expedite appraisal activities, and provide valuable feedback on factual

accuracy.

Relations between the appraisal team and representatives from the assessed organization must be cordial,

open, and professional. It is in the interest of both EA-22 and the local representatives to approach security

appraisals in partnership to ensure that these activities lead to potentially useful and effective program

improvements. Positive relationships are especially important in performance testing, where the trusted

agents are relied on to maximize realism while maintaining the confidentiality of the scenario or test content

and the timing of the scheduled and limited-notice tests. EA keeps the number of trusted agents as low as

possible and marks all performance test documents (including knowledge tests) with a header or disclaimer

that reads: “Trusted Agent Use Only, Not to be Disseminated.” All trusted agents also sign EA’s

Memorandum of Understanding and Agreement Regarding Trusted Agent Duties and Responsibilities form

before being briefed on sensitive test information. Finally, EA shares performance test materials with

trusted agents only in person or, when necessary, by encrypted emails.


January 2017 9

Assessment Plan Development

For each appraisal, EA-22 develops an assessment plan that includes scope, schedule, team responsibilities

and assignments, and process. The assessment plan also provides a topic schedule of appraisal activities

(data collection, performance testing, analysis of results, and report development), team composition, and

lines of inquiry. See Appendix C, Example of a Multi-Topic Assessment Plan, for more detail. Lines of

inquiry are derived from policy requirements (must, shall, will), with a focus on elements identified during

the scoping process. The lines of inquiry help topic teams better understand how the site’s protection

programs implement S&S policy, site procedures, and Federal regulations.

Planning Meetings

Two separate but concurrent planning meetings take place during pre-appraisal activities. One brings the

topic teams together, usually at Headquarters but sometimes at a field site, to review the data call, begin

identifying potential topic strengths and weaknesses, validate the topic scope, select and schedule data

collection methods, and share information with Office of the Associate Under Secretary for Environment,

Health, Safety and Security (on potential policy concerns) and the National Training Center (NTC) (onsite

training history and concerns). Topic teams often identify additional data call and support requirements

during this meeting and communicate these to their site POCs. The protective force topic team is most

likely to conduct part of its planning meeting at the site due to the complexity of data collection activities

in this topical area and the benefits gained from being on site. In exceptional cases, site, program office,

and EA senior managers may approve some limited onsite data collection activities during the planning

meeting.

The other planning meeting, conducted on site by EA FOF test planners, focuses on meeting with trusted

agents, adjusting test scenarios, receiving briefings, conducting safety walkdowns, and developing safety

risk assessments. The other major meeting objectives are determining the rules of engagement, identifying

“out-of-plan” areas, finalizing the test schedule and scenarios, and obtaining trusted agent concurrence on

test plans. Toward the end of the meeting, the EA performance test director contacts the Composite

Adversary Team Coordinator to discuss the number of adversary team members required for the test and

the Engagement Simulation Systems Support Coordinator to determine the types and quantities of

equipment needed for the test.

2.2 Appraisal Conduct

Data Collection

The topic teams collect data while on site, using techniques that typically include document reviews, tours

and observations, interviews, knowledge tests, and performance testing. All data collection techniques

exhibit elements of compliance and performance, and the full spectrum of techniques is necessary for an

effective appraisal. The goal in appraisal conduct is to accomplish all planned data collection activities

fairly, impartially, and professionally and to validate the technical accuracy of the collected data. The

Independent Oversight Program Appraisal Process Protocols provide additional information on data

collection techniques.

Validation

EA uses a continuous validation process to ensure the factual accuracy of the collected data and to

communicate any identified deficiencies, and their impacts, to the responsible managers and organizations.


10 January 2017

Topic teams validate data collection results (favorable and unfavorable) as soon as they occur, at the end

of the day or week, and near the end of appraisal execution. This continuous process verifies that the

collected data is factually correct and can legitimately be used to evaluate the effectiveness of the program,

project, or activity. It also provides an opportunity for the assessed site to provide additional data that may

mitigate any emerging concerns. The Team Chief and Deputy Team Chief validate the previous day’s

information in morning meetings with field Federal and site S&S contractor managers, described further

below.

Data Analysis

Although EA-22 actually starts analyzing data before the onsite visit begins, the extensive data collected

on site allows a greater focus on emerging performance and compliance trends. Data analysis can also

generate additional data collection activities and validation discussions with site security officials. The

sooner data analysis can be validated with the site, the better.

Topic teams use a team approach vetting each strength and weakness. Appendix F provides a tool to help

topic teams determine whether a weakness is best characterized as a deficiency or a finding.

End-of-Day Team Meetings and Morning Manager’s Meetings

End-of-day team meetings are an opportunity to communicate and integrate topic team data collection

results and to highlight key points and/or potential concerns for the appraisal team management’s

awareness. Each topic team prepares and briefs a list of narrative bullets summarizing data collection

activities and results for the day. After the conclusion of the end of day meeting, senior EA managers

and staff are provided the final version of the bullets. When briefing team management, Topic Team

Leaders indicate which of their topic’s summary bullets were validated with the site. This critical action

ensures that the Team Chief is aware of validated information during the next day’s morning manager’s

meeting. The Team Chief and Deputy Team Chief discuss the previous day’s information in morning

meetings with field Federal and site S&S contractor managers.

The primary purposes of the morning manager’s meeting are to openly communicate the progress of

appraisal activities and potential concerns, and to verify that the appraisal team’s information and the site’s

information regarding key data collection results are consistent. Another important objective of this

meeting is to discuss the day’s data collection activities and follow-on meetings. These meetings ensure

that site POCs and their managers are aware of the data that has been collected; the intention is for the site

to acknowledge factual accuracy, provide corrected information, request further data collection, or provide

mitigating information.

Report Writing

In the report writing phase, the topic teams thoroughly analyze the data, develop conclusions, and, based

on the analysis and conclusions, prepare a report that accurately reflects the status of the program, program

element, facility, or activity that was examined. For larger multi-topic assessments, report writing typically

begins mid-week of the second onsite data collection trip and continues through the remainder of the

appraisal.

Appraisal reports include a one-page summary, executive summary, introduction, and topical appendices.

The principal writer is responsible for the one-page summary, executive summary, and introduction. The

one-page summary is of special importance because it communicates the overall results of the appraisal to


January 2017 11

the Secretary, other senior DOE leaders, and other interested external stakeholders, such as Congress. To

facilitate communication of the results, the one-page summary typically contains only unclassified or

controlled unclassified information.

The topic team writers support the Topic Team Leader and are responsible for drafting their topic’s

appendix. Each topical appendix consists of an introduction; the identified results, best practices, findings,

and deficiencies (when applicable); conclusions; and opportunities for improvement. Within topical

appendices, the topic team must exert care to communicate all relevant results and the status of program

effectiveness, all in support of the conclusions. A key report element is identifying both best practices and

topic weaknesses and determining whether the weaknesses are deficiencies or findings (warranting high-

level management attention), as described in Appendix F. EA-22 normally assigns a finding for significant,

systemic, or recurring/chronic deficiencies that have a demonstrated or highly likely negative impact on the

protection of high consequence security assets. The final section of an appendix is opportunities for

improvement, which informs the assessed site of possible ways to improve program performance in areas

where findings have been identified. Reports may also provide recommendations (focused on management

systems rather than a specific deficiency) and best practices observed during an assessment that merit

consideration by other DOE/NNSA and contractor organizations.

The EA process ensures that all report elements are thoroughly reviewed before transmittal of the draft

report to site personnel. Topic team reviews are the first step in this phase of developing the report. After

topic teams have generated a draft document, the Management Review Board (consisting of the Team

Chief, Deputy Team Chief, and technical advisor, and other EA-22 managers as desired) reviews it to ensure

that all appendices are consistent with each other and that each appendix is readable, logical, and adequate

to support the conclusions. Subsequently, the QRB reviews all elements of the report. The QRB, composed

of senior EA managers and senior personnel (usually former senior DOE managers) who support EA,

provides comments that result in appendices undergoing a second review to ensure the report meets EA’s

high quality standards.

NTC representatives also participate as observers in QRB deliberations to identify findings and

recommendations that may have training implications for the NTC. During this phase of an appraisal, NTC

representatives note QRB participants’ observations, findings, and opportunities for improvement involving

training concerns, and also meet informally with team members to gain added insights into potential training

concerns.

Following the QRB review, the EA-22 Team Chief schedules a factual accuracy review with the site’s

security management team to help identify and resolve potential concerns about the draft report. These

reviews typically last no more than four hours and generate initial site comments that may be accepted,

accepted with modification, or not accepted by EA-22 management. Site management is informed of the

resolution of the comments before the closeout briefing (described below). The objective of the factual

accuracy review and comment period is to ensure that EA’s draft report is factually correct and presents

an accurate evaluation of the effectiveness of the facility’s protection program.

Exit Meeting (on the last day of onsite assessment activity)

The final step is the onsite exit meeting, which includes a closeout briefing. The EA-22 Team Chief

typically conducts this briefing, using presentation slides, to convey the team’s observations and

preliminary results to the field Federal and contractor managers. Topic Team Leaders and team members

attend the closeout briefing as needed or requested. At the end of the closeout briefing, EA-22 formally

provides a copy of the draft report to the responsible DOE/NNSA field office manager to begin the ten-day

factual accuracy review (see Section 2.3). Either before or after the closeout briefing, the EA-22 Director


12 January 2017

or Deputy Director, or a senior EA manager not assigned to the assessment team, meets with senior site

line managers and security officials to obtain their feedback on the appraisal process.

2.3 Post-Appraisal Activities Post-Appraisal Validation Review

EA-22 formally provides a copy of the draft report to the applicable Headquarters program office

representative immediately after the assessment team returns to Germantown Headquarters. The site and

program office have ten calendar days (beginning on the day of the onsite closeout briefing) to complete

their post-appraisal review of the report and provide factual accuracy comments to EA-22; EA-22 asks the

program office to consolidate all comments before forwarding them. EA-22 then reviews each comment

and develops a response. EA-22 typically accepts the comments as provided, accepts them with

modification, or declines to accept them due to technical accuracy. EA provides an explanation for any

comment that is not accepted and coordinates additional meetings and/or teleconferences as required to

complete resolution of the comments. EA also reviews the comment resolution comments with the program

office, either concurrently with or separately from site line management. If disagreements on comment

resolution between the EA Office Director and the applicable Headquarters program office representative

persist, they proceed to higher management levels, up to and including the Deputy Secretary, for resolution.

For multi-topic assessments, the cognizant Program Secretarial Officer has the opportunity to submit a

written management response to the draft report’s conclusions and recommendations. If a Secretarial

Officer provides a response, EA reflects this information in an appendix to the report.

Comment Resolution and Technical Edit

The Team Chief and Deputy Team Chief, working with Topic Team Leaders, resolve all program office

comments, incorporate needed changes into the draft report, and provide a summary of the rationale for any

rejected comment to the program office. The draft report then undergoes a technical edit, typically via the

classified network before becoming final.

Final Report

Within 60 days after the exit meeting, the report is submitted to the EA Office Director and undergoes a

final review before being released to the Office of the Secretary for approval. The approval process includes

briefing the Secretary’s staff via a one-page summary of the report. Once approved, final reports are

uploaded into the Safeguards and Security Information Management System.

Briefings

EA may conduct briefings for key Headquarters managers before or after the final report is approved.

External briefings to key stakeholders, such as Congressional staff and the Defense Nuclear Facilities Safety

Board, are conducted, if appropriate, only after issuance of the final report and completion of all internal

DOE briefings.


January 2017 13

Sharing Lessons Learned

EA-22 conducts quarterly teleconferences with Headquarters and field security officials to share lessons

learned from recent appraisals. Lessons learned are also routinely briefed at meetings of senior Federal and

contractor security directors and during various DOE working group meetings, and EA-22 prepares an

annual report identifying S&S trends and lessons learned. The NTC and the Office of the Associate Under

Secretary for Environment, Health, Safety and Security regularly receive information on lessons learned

from EA-22 as well.

Office of Security Assessments Appendix A Appraisal Process Protocols

A-2 January 2017

CA

T =

Com

posi

te A

dver

sary

Tea

m

C-L

AN

= C

lass

ifie

d L

oca

l A

rea

Net

work

E

A =

Off

ice

of

En

terp

rise

Ass

essm

ents

E

A-2

2 =

Off

ice

of

Sec

uri

ty A

sses

smen

ts

ES

S =

En

gag

emen

t S

imu

lati

on

Syst

ems

FO

F =

Forc

e on

Fo

rce

IT =

In

form

atio

n T

ech

nolo

gy

L

SP

T =

Lim

ited

-Sco

pe

Per

form

ance

Tes

t

NT

C =

Nat

ional

Tra

inin

g C

ente

r P

OC

= P

oin

t of

Con

tact

S

SIM

S =

Saf

egu

ard

s an

d S

ecu

rity

In

form

atio

n M

anag

emen

t S

yst

em

Mu

lti-

To

pic

Ass

essm

ent

Tim

elin

e

Appendix A – Multi-Topic Assessment Timeline

Office of Security Assessments Appraisal Process Protocols Appendix B

January 2017 B-1

Appendix B – Example Document Request (Data Call)

The following is a template for each topical area and should be revised according to the specific scope for

each Office of Enterprise Assessments (EA) appraisal conducted by the EA Office of Security Assessments

(EA-22). Topic Team Leaders must coordinate to ensure that the same documents are not requested in

multiple topical areas unless operationally necessary.

Protection Program Management (PPM)

The following items are normally provided to Headquarters to support planning meeting activities.

Safeguards and Security (S&S) Management and Administration

1) Organizational charts with names for all Federal and contractor site elements (including significant

subcontractors) that have S&S responsibilities.

2) Federal field office and contractor missions and function manuals and/or other reference material

describing the roles and responsibilities of current site organizations, including deliverables and

accountability within the S&S program.

3) Fiscal year (FY) (insert date range) Performance Evaluation Report(s) and FY (insert date range)

Performance Evaluation Plans for each prime contractor.

4) Current Federal and contractor delegation of authority memoranda.

5) Program office guidance and local instructions for the implementation of S&S programs.

6) Federal field office procedures for incorporating new/revised U.S. Department of Energy (DOE)

directives into the contract(s).

7) List of requirements documents in the current contracts with S&S responsibilities (e.g.,

management and operating contractor, S&S services support contracts).

8) Implementation plans for any S&S-related directives not fully implemented.

9) Approved and submitted equivalencies/exemptions (deviations), including justification, risk

analysis, and approvals as applicable.

10) Federal and contractor procedures/process documents for equivalency/exemption (deviation)

submissions and approval.

11) FY (insert date range) Annual Training Plan(s) for Federal and contractor S&S staff (non-

protective force).

S&S Planning

12) List of all Federal and contractor entities with a facility clearance supporting the site’s operations.

13) Current approved Federal, contractor, and any other possessing entity facility/site security plans

(SSPs).

14) Current vulnerability assessment (VA) report (VAR) supporting the most current approved SSP,

and the associated VA procedure.

15) The most current analytical basis that addresses the security assets present at the site, including

(but not limited to) hazardous materials and special nuclear material (SNM) rollup.

16) Compensatory measures and approvals as applicable.

17) Security Incident Response Plan and Tactical Doctrine implementing documents.

18) Security Conditions (SECON) Plan(s) and supporting implementing documents (e.g., procedures,

checklists).

19) Regional and local threat assessments.

20) Current applicable memoranda of understanding (MOUs) and memoranda of agreement (MOAs)

with local, state, and Federal law enforcement agencies.

Office of Security Assessments Appendix B Appraisal Process Protocols

B-2 January 2017

21) Federal and contractor procedure/process for controlling changes/updates in relevant documents

(e.g., plans, procedures).

S&S Program Assurance

22) Federal survey/assessment reports and contractor self-assessment reports for FY (insert date

range).

23) Guidance on Federal oversight, survey procedures, FY (insert date range) contractor self-

assessment procedures, and FY (insert date range) schedules.

24) Federal and contractor procedures for resolution of findings (external inspections, surveys and

self-assessments, observations, and/or other action items related to the mitigation of identified

weaknesses in the S&S program).

25) Federal and contractor corrective action plans for previous EA appraisals.

26) Records, other than those in the DOE Safeguards and Security Information Management System,

that reflect Federal verification, validation, and closure of deficiencies, findings, concerns, and/or

observations for FY (insert date range).

27) Performance Assurance Program (PAP) Plan and listing of essential elements.

28) Most current essential element test plans.

The following items are typically available in the PPM team workspace when they arrive at the site.

29) Briefings/discussions by Federal/contractor personnel (as coordinated and scheduled with the

EA-22 PPM Topic Team Leader) that include:

a) Overview of contract management structure.

b) Description of changes in program management since the last EA appraisal.

c) Overview of the VA and SSP development processes, including:

i. Developing adversary strategies/tactics.

ii. Evaluating insider adversaries, both working alone and in collusion with the outsider.

iii. Conducting computer modeling/performance testing analyses and developing

neutralization values.

iv. Developing and evaluating upgrade/efficiency packages.

v. Chronological efforts and events leading up to the current/approved SSP (including VAs,

validations, peer reviews, Headquarters visits, etc.).

vi. Mechanisms for performance-tested validation of VA assumptions and values.

d) Overview of the Federal and contractor survey/self-assessment programs, including the process

for conducting risk-based evaluations.

e) Overview of the corrective action/issue management and causal analysis processes to prevent

recurrence of deficiencies.

f) Summary of the status of corrective actions for prior EA findings.

30) Data from performance testing that supports the most recent/current VA or performance

assumptions in the VAR.

31) Files used to develop the VA and the associated evidence files for the following types of data:

a) Modeling inputs.

b) Protective force response.

c) Adversary capabilities.

d) Blast effects.

e) Sabotage data (if appropriate).

f) Timeline data.

g) Neutralization data.

h) Special weapons effectiveness.

32) Simulation reports and capability to review sample scenarios.

33) FY (insert date range) PAP test reports for all topical areas.


January 2017 B-3

34) Evidence files supporting Federal surveys and contractor self-assessments for FY (insert date

range).

35) Federal and contractor qualifications and training records for S&S staff (non-protective force).

Physical Security Systems (PSS)


1) A site plan drawing (11"x17" or larger) indicating security areas and target locations associated

with the site. Include maintenance and communications facilities and the locations of the central

alarm station (CAS) and secondary alarm station (SAS). Include a separate drawing (11"x17" or

larger) for the Protected Area and perimeter intrusion detection and assessment system zones.

2) Compensatory plans associated with PSS equipment that is out of service.

3) Description of explosive detection program.

4) Copies of approved/pending PSS-related deviations.

5) List of PSS-related documents approved by the Officially Designated Security Authority.

6) List of PSS-related documents approved by the Officially Designated Federal Security Authority.

7) Copies of PSS-related self-assessments completed during the past 12 months.

8) Copies of PSS-related surveys completed during the past 12 months.

9) Copies of corrective actions associated with findings resulting from the previous EA assessment.

10) Information system security plans (or other terminology, such as cyber security plans) for

computer-controlled PSS.

11) List of privileged users (job title only) for all computer-controlled PSS.

Alarm Management and Control Systems (CAS/SAS operations, auxiliary power)

12) System descriptions and diagrams for alarm monitoring and control systems. For alarm

monitoring systems, schematic diagrams should include locations of major equipment (data

gathering panels, activation panels, and communication panels) for interior and exterior systems.

13) System descriptions and diagrams for primary and auxiliary power systems associated with PSS,

including generators, automatic failover systems, uninterruptible power supply systems, etc.

14) Procedures for testing auxiliary power systems, and documentation indicating the results of tests

conducted during the past year.

15) Description of all types of line supervision and tamper alarms in use, and their locations.

16) Procedures for conducting CAS/SAS failover tests, and documentation indicating the results of

tests conducted during the past year.

17) Configuration management policies and procedures for PSS hardware and software.

18) Alarm system operator training and testing documentation.

19) List of all vaults, Closed Areas, and/or vault-type rooms (VTRs).

Communications (radio communications, alternate communications)

20) System descriptions and diagrams for security-related voice communications, which should

include a VA. A description of the emergency communications plan (may be part of the Tactical

Defense Plan or Protective Force Response Plan).

Barriers

21) Descriptions and diagrams for active or passive barrier systems used to direct or control the

movement of personnel and vehicles through security area boundaries.

22) Procedures for identifying potential barrier degradation.


B-4 January 2017

Access Control Systems, Entry/Exit Inspections, and Badging

23) System descriptions and diagrams for access control systems, including interfaces between the

badge system, access control system, and alarm monitoring system, and the procedures for badge

fabrication and personnel enrollment. A VA addressing these interfaces should be included. Also

provide specific procedures for removing or changing access authorization in the access control

system for individuals whose need for authorization has changed, and destruction procedures for

badges returned to the badge office.

24) Description of screening procedures used at security boundaries to identify prohibited and/or

controlled articles.

Testing and Maintenance and PAP

25) Procedures and program description for false/nuisance alarm rate review, analysis, and corrective

action development.

26) False/nuisance alarm rate data and analysis for security system sensors for the most recent six

months available.

27) Security systems preventive and corrective maintenance and testing program documentation and

procedures. List of maintenance activities associated with PSS within the last 12-month period.

28) Performance testing criteria for alarm monitoring and video assessment systems, access control

systems, and security-related voice communications systems.

29) List of PSS-related performance testing activities completed during the past 12 months.

30) Acceptance testing procedures for installation of new or replaced security systems.

Lock and Key Program

31) Program documentation and associated procedures related to security locks and keys. Also

provide security key accountability documentation and audit results generated during the past

year.

System Upgrades

32) Description of new and/or upgraded systems installed and declared operational since the last EA

assessment.

33) Description of planned upgrades relating to PSS, including status of authorization and funding, as

well as the expected date the planned upgrades will be in service. Also provide descriptions and

dates of implementation for upgrades and modifications to alarm monitoring and video assessment

systems, access control systems, and security-related voice communications systems completed

in the last two years (hardware and software), and the acceptance testing documentation associated

with these modifications and upgrades.

Protective Force


In addition to the requested items below, EA will require access to protective force training records once

onsite data collection begins.

Management Documentation 1) Current protective force contract, including all applicable modifications.

2) List of DOE orders cited in the protective force contract and any applicable implementation plans.

3) Latest approved SSP.

4) VAR and/or risk analysis report.

5) Latest Performance Assurance Plan, if separate from the SSP.

6) Latest approved SECON Plan, if separate from the SSP.


January 2017 B-5

7) Delegation letters for the Cognizant Security Office, the Officially Designated Federal Security

Authority, and any applicable Officially Designated Security Authority (Federal or contractor).

8) Any current or pending deviations (equivalencies/exemptions) from DOE S&S directives.

9) Supervisor post/patrol duty check logs for the (insert date range) time period.

10) Armorer maintenance records for the last two cycles (one-year period).

Protective Force Survey and Self-Assessment Documentation

11) Most recent Federal survey report.

12) Most recent protective force self-assessment report.

13) Corrective action plans, including the associated evidence files for any open findings resulting

from the most recent survey and self-assessment. (Note: If evidence files are too cumbersome to

scan, EA-22 can be review them on site during assessment activities.)

Protective Force Plans and Contracts

14) Security Incident Response Plans.

15) SECON implementing procedures or post orders.

16) Protective force work stoppage contingency plan.

17) Site map indicating protective force fixed posts and patrol areas.

18) Current and projected protective force strength report, including figures for the last four quarters.

19) Current protective force contract, including any modifications since the contract was originally

awarded.

20) Local law enforcement authority, including implementation/re-certification plans.

Protective Force Orders and Procedures

21) Protective force organization chart.

22) Protective force general orders.

23) Protective force post orders.

24) Protective force supervisory orders if separate from post/patrol orders.

25) Protective Force standard operating procedures.

26) A current “on-shift” and “off-shift” protective force duty roster that includes shift schedules,

limited-duty rosters, and vacation rosters for the period (insert date range). Note: If the site has

not yet developed a duty roster/schedule for the requested period, provide the most current one

available.

27) 10 CFR 1046 implementation plan, including the status of its overall execution.

28) Use-of-force policy.

29) Rules of engagement.

30) Range safety procedures.

Protective Force Memoranda of Understanding or Agreement

31) Current MOU/MOA, showing all required signatures, with law enforcement, emergency

services, and any other support organizations.

Protective Force Training, Testing, and Assessment Plans

32) Protective force annual training plan.

33) Protective force training approval program certification.

34) Protective force performance testing plan.

35) Four to eight performance test plans for protective force essential elements:

a) SNM detection.

b) Entry/exit portal detection.

c) Explosive detection.


B-6 January 2017

d) Active shooter.

e) Response to unmanned aerial system.

36) Protective force issues management protocol. (Note: Once on site, EA-22 will request test plans

for specific assessment activities.)

Equipment

37) Protective force inventory of equipment, including vehicles, weapons, radios, and non-lethal

weapons and identifying the locations and types of individual post/patrol assigned equipment.

The following protective force documents should be available upon request for review.

38) Individual security police officer (SPO) qualification records (weapons, physical fitness).

39) Certification records for instructors (classroom and range), armorer, and CAS operators.

40) Individual SPO training records.

41) SPO lesson plans.

42) Sample of supervisor and post logs.

43) Target folders.

44) Description of canine program and training records, schedules, and results associated with

performance testing.

45) Full color photographs (8.5"x11") of SNM that is currently being protected, stored, and/or

processed on site.

46) Vehicle maintenance service records.

Material Control and Accountability (MC&A)


1) The current approved MC&A Plan with the approval letter from the site office.

2) A draft MC&A Plan if the site anticipates submitting a revised version before EA’s onsite

assessment.

3) All approved material balance area (MBA)-specific MC&A Plans as applicable.

4) MC&A procedures manual.

5) List of approved MC&A deviations with supporting documentation.

6) MC&A documentation used to support the Category of irradiated fuel, including procedures for

monitoring rollup.

7) Site office MC&A survey reports for surveys conducted in (insert date range).

8) List of approved tamper-indicating devices (TIDs) and TID custodians, applicators, and/or TID

verifiers, along with TID procedures (if not included in MC&A procedures manual).

9) Current shipper/receiver agreements.

10) Emergency response plans pertinent to the loss of control of SNM.

11) If temporary material access areas (MAAs) are used, the procedures in effect when temporary

MAAs are established.

12) All facility-reported incidents of security concern (IOSC) involving MC&A, security, and

operations for the past two years with backup documentation.

13) MC&A Training Plans, MC&A Performance Testing Plan, and Measurement Qualification Plans

(these are distinct from procedures but may be included in other documentation).

14) Training records (electronic) for all personnel performing MC&A functions.

15) Complete list of findings and corrective actions from DOE surveys and internal reviews for the

past two years.

16) Internal Review and Assessment Plan and procedures, list of internal reviews and assessments

completed since (insert date).


January 2017 B-7

17) A current list of MBAs and MBA custodians and alternates.

18) Plans and procedures for measuring holdup (if not included in the MC&A procedures manual).

19) Safeguards termination documentation on nuclear materials terminated in (insert date range).

20) Dates of Nuclear Materials Management and Safeguards System annual submissions for (insert

date range).

21) Summary of shipper/receiver differences identified since (insert date).

22) List of current measurement procedures for approved accountability systems.

23) List of current measurement control procedures for approved accountability systems.

24) Inventory procedures and schedules (if not included in the MC&A procedures manual).

25) If Comprehensive Analyses of Safeguards Strategies (COMPASS) software is used, results of the

two most recent COMPASS analyses.

26) VA demonstrating MC&A analyses for Category I SNM locations and other locations where

rollup to Category II is credible.

27) List of any MC&A performance incentives/award fee items in the MC&A topical area.

The following information should be readily available from site MC&A personnel.

28) Summary inventory list for MBAs, showing total quantities for each material type (may be kept

with accounting organization and will be reviewed during the assessment).

29) Material balance reports (M-742) since (insert date).

30) Inventory differences, limit of error of the inventory difference, and key measurement error

contributors by material type, by MBA, for each inventory period for the last 24 months.

31) Qualification reports for all instruments currently in use for accountability measurements.

32) Verification and accountability measurement results, including resolution paths for items that

failed verification.

33) List of replicate data used in the determination of measurement uncertainties for the last six

inventories.

34) The results of process and item monitoring (for all MBAs that employ these activities) for the past

24 months.

35) Summary information on key holdup points in the facility and holdup measurements conducted

since (insert date).

36) Status of SNM considered unmeasured and/or not amenable to measurement, by MBA.

37) Sampling plans (if not included in the MC&A procedures manual) and the results of the

verification/confirmation program for the previous two physical inventories.

In addition to the data listed above, please prepare a briefing (to be given on the first day of the assessment,

after inbriefing and initial orientations) that provides the following information:

A description of the current MC&A organizational structure (Federal and contractor, including the

names of individuals and funding lists).

Any changes to the MC&A system and the operational status of the facility (including any process

activities that may have changed the characteristics of existing material types or produced new

material types) since (insert date).

An overview of planned activities/production schedule for MC&A during the next two years (e.g.,

mixed oxides, processing of Category I and II quantities), along with any staffing/funding concerns.

Summary/status of any ongoing and proposed termination of safeguards activities (e.g., plutonium

downblending).


B-8 January 2017

It is very important that the briefing also include the results of recent assessments and key issues currently

being addressed by the MC&A program. Copies of the briefing notes should be provided to the assessors.

Classified Matter Protection and Control (CMPC)


1) A list of vaults, VTRs, and Closed Areas (by building and room number) that currently store any

level (Confidential, Secret, Top Secret) and category (e.g., National Security Information,

Restricted Data, Weapons Data, North Atlantic Treaty Organization, Foreign Government

Information) of classified matter, regardless of form (documents, electronic media, parts, etc.).

2) A list of classified holdings that are stored in non-standard storage configurations, and the

compensatory measures for each location.

3) Copies of the security plans for non-conforming or non-standard storage areas, including the

Officially Designated Federal Security Authority’s approvals for the areas.

4) A list, by organization, of the custodians or persons responsible for the classified repositories,

vaults, VTRs, and Closed Areas listed above that have accountable items (Top Secret, United

Kingdom, North Atlantic Treaty Organization, Sigma 14, Sigma 20 media, and Secret/Restricted

Data stored outside of a Limited Area, or as non-accountable), including phone numbers and

locations (building and room number).

5) A current inventory listing of all accountable classified matter, regardless of form (documents,

electronic media, parts, etc.) and their storage locations or by account.

6) Copies of the last two annual inventories.

7) Copies of local procedures, self-assessments, and equivalencies/exemptions (or deviations

pending conversion to equivalences/exemptions) pertaining to the following programs/areas:

a) Operations security (OPSEC)

b) CMPC, including training material

c) Hand-carry, including briefing material

d) Classified matter destruction, reproduction, and transmission

e) Mailrooms

f) Technical surveillance countermeasures (TSCM).

8) List of points of contacts with phone numbers for the programs listed above, including both

Federal and contractor personnel.

9) Copies of the corrective action plans for the last EA assessment of CMPC.

10) Copy of the classified matter custodian training program.

11) Copies of the OPSEC Plan and site critical information.

12) Copy of the most recent (within the past 36 months) OPSEC assessments or reviews.

For items 13 and 14 below, include these only if the items are unclassified. If they are classified, EA-22

will review them on site.

13) Copy of local policy and procedures for the TSCM program and team.

14) Copies of training and briefing materials used to train TSCM and site personnel.

The following items should be made available for review at the site during the week of (insert date). Since

these items are normally classified, the assessors expect to conduct this portion of the assessment in the

space where these items are normally stored/used.

15) Copies of the last five years of reports of TSCM services.

16) Information on the scheduling of TSCM services.

17) Last self-assessment of the TSCM program at the site.


January 2017 B-9

18) Information concerning lifecycle replacement of TSCM equipment.

Incidents of Security Concern

19) Copy of the IOSC program plan and documentation of approval.

20) IOSC-related procedures.

21) List of all Federal and contractor IOSC incidents in FY (insert date range), sorted by category and

incident type and including local tracking numbers and resolution actions.

22) Copy of IOSC damage assessments for FY (insert date range).

23) Copy/access to IOSC awareness training materials.

24) Copy of the inquiry official’s qualification summary and associated training records.

25) Copy of the inquiry official’s appointment/delegation letter.

26) Copy of foreign ownership, control or influence (FOCI)/facility clearance (FCL) procedures,

including the process/procedure for reporting changes that may affect the facility clearance level.

27) Copy of documentation for FCL approvals on current contractors and tier parents for which the

site has cognizance.

28) A list of all “Possessing” facilities for which the site has cognizance, and their facility codes.

29) A list of cleared personnel within each of the facilities:

a) Identify any contractors under FOCI mitigation and provide documentation for the trustees,

proxy holders, and/or outside directors in connection with the mitigation plan.

b) For contractors under mitigation, provide certification that no changes have occurred in their

ability to protect classified information.

c) Copy of documentation of annual reviews and, for contractors under mitigation, certification

that no changes have occurred in their ability to protect classified information.

d) Copy of documentation pertaining to FCL suspensions related to FOCI mitigation.

Human Reliability Program, Personnel Security, and Unclassified Foreign Visits and Assignments

30) An organization chart(s) or other means of describing the structure supporting the overall

personnel security program, sufficient to show where all key program officials and support staff

reside organizationally and the chain of command to each key program official and support staff.

Human Reliability Program (HRP)

31) Copy of the HRP implementation plan(s) and documentation of review and approval.

32) List of positions identified for HRP certification and a description of the process for evaluating

these positions for certification, including those positions applicable to 10 CFR 712 Category 4.

33) List of all Federal and contractor employees enrolled/certified in the HRP program as of (insert

date).

34) A separate, alphabetized (last name first) list for each of the following for the period of (insert

date range), as follows:

a) All HRP individuals who have been temporarily removed, with the date and the reason for

temporary removal (security, safety, medical, or change of position/employment).

b) All HRP individuals who have had their HRP certification revoked, with the date and the reason

for revocation (security, safety, medical, or change of position/ employment).

c) All individuals denied HRP certification by the site Certifying Official.

d) All HRP individuals who have had any disciplinary action(s), including the reason for the

disciplinary action, the date the disciplinary action was taken, and whether the individual was

temporarily removed from HRP because of disciplinary action.

e) All HRP individuals who have been involved in an accident or incident, including IOSCs

reported to DOE as an occurrence.


B-10 January 2017

f) All non-HRP escorted individuals, indicating the dates of entry, grouped for each escorted

visitor and including all dates (most recent first) of entry for each MAA accessed; the

individual’s employer; and authorized clearance level at the time of entry (Q or L).

35) A list of job titles for each organization where job task analyses have been developed and an

example (blank form) of the format used by each organization.

36) Access to HRP files and HRP-associated medical and psychologist files (make available during

onsite visits).

37) Copy of HRP training and instructional materials (computer-based training, classroom, etc.).

38) Copy of designation letters for the following positions (as applicable):

a) HRP Certifying Official.

b) HRP Management Official(s).

c) Designated Physician.

d) Designated Psychologist.

Personnel Security

39) Separate, alphabetized (last name first) lists of the following for the site contractor:

a) A list of personnel with an access authorization. The timeframe for all lists is (insert date

range).

b) A list of all completed pre-employment checks.

c) A list of all individuals who had absences of 90 days or more. (Payroll records should be used

to develop this list.)

d) A list of all contractor/subcontractor employees for whom the site has notified or reported to

the DOE personnel security organization information of personnel security interest as a result

of a disciplinary action (omit security infraction reports of the sort requested by the CMPC

topic team), the reason for the disciplinary action, and the date reported to the DOE personnel

security organization.

40) A separate, alphabetized (last name first) list of the following for contractor employees with a Q

access authorization for the period of (insert date range):

a) A list of access authorization terminations and the date (omit transfers or anything that is not a

termination of the access authorization).

b) A list of all employment terminations and the date (omit individuals who were re-employed by

the site or a subcontractor within six months).

c) A list of all individuals who have been granted an initial access authorization, the date DOE

took the action to grant, and the date a DOE security badge was issued.

d) A list of all individuals who have transferred from another DOE facility, the date DOE took

the action to transfer, and the date a DOE security badge was issued.

e) A list of all individuals who have had their access authorization reinstated, the date DOE took

the action to reinstate, and the date a DOE security badge was issued.

41) Procedure/description of pre-employment and annual random drug testing for Federal personnel,

contractors, and subcontractors with access authorizations or those requesting an access

authorization. Testing of these individuals is required by Secretarial memorandum dated

September 14, 2007.

42) List of cleared contractor employees (including subcontractors) whose workspace as of (insert

date) is located outside of a Limited, security/exclusion, or Protected Area. This list should

include offsite buildings/facilities only if there are more than ten cleared employees in the

workspaces.

43) Copy of the current initial, comprehensive, and termination briefings and the most recent annual

security refresher briefing.


January 2017 B-11

Unclassified Foreign Visits and Assignments (UFV&A)

44) The total number of unclassified foreign national visitors and assignees who visited the site from

(insert date range), broken out in the following categories: non-sensitive country foreign nationals,

sensitive country foreign nationals, sensitive subjects, and access to security areas (Limited Areas,

Protected Areas, MAAs).

45) A separate, alphabetized (last name first) list of UFV&As from (insert date range) for items a

through j below. Each of these lists should provide the following information: name and

nationality of visitor/assignee, date of visit/assignment, name of host/escorts, facilities included in

the scope of the visit/assignment, and, when applicable, approval for remote or onsite access to

computing systems.

a) Foreign nationals from sensitive countries.

b) Sensitive subjects.

c) Access to an MAA, Protected Area, Limited Area, or security/exclusion area.

d) Foreign nationals from terrorist countries.

e) Unescorted access to any site security area (Limited Area, Protected Area, or MAA).

f) Access to site computing assets, including a termination date for access to the computing assets.

g) Remote access to site computing assets, including a termination date for remote access.

h) After-duty-hours access to any site facility.

i) Incidents and inquiries of security concern.

j) The most frequently visited site facilities (building or areas) and program organizations.

46) A diagram showing all site security areas (MAAs, Protected Areas, Limited Areas), including

building numbers. (This diagram may be a duplicate of those provided to other topic teams.)

47) Procedures and/or protocols used to process and approve all UFV&As.

48) An example of a generic security plan for a foreign visit or assignment and a copy of a specific

security plan.

49) Copy of host/escort guidance, or training materials.

50) Procedure or description of how lessons learned are shared with other hosts/escorts.

Office of Security Assessments Appendix C Appraisal Process Protocols

C-1 January 2017

Appendix C – Example of Multi-Topic Assessment Plan

OFFICE OF ENTERPRISE ASSESSMENTS PLAN

FOR CONDUCTING SAFEGUARDS AND SECURITY ASSESSMENT AT THE

(Facility Name)

(DATE)

Office of Enterprise Assessments

U.S. Department of Energy

Office of Security Assessments Appraisal Process Protocols Appendix C

January 2017 C-2


FOR CONDUCTING A SAFEGUARDS AND SECURITY ASSESSMENT AT THE

(Facility Name)

(DATE)

Approved by: ______________________________________ Date: ________________

(Name) Director

Office of Security Assessments


Approved by: ______________________________________ Date: ________________

(Name) Director



Approved by: ______________________________________ Date: ________________

(Name) Deputy Director



C-3 January 2017



(Facility Name)

(DATE)

PLAN CONTENTS

Assessment Plan and Approach ........................................................................................................ C-4

Attachment A: Assessment Schedule .............................................................................................. C-9

Attachment B: Assessment Team Composition .............................................................................. C-10

Attachment C: Protection Program Management Lines of Inquiry ................................................. C-11

Attachment D: Physical Security Systems Lines of Inquiry ............................................................ C-12

Attachment E: Protective Force Lines of Inquiry ............................................................................ C-14

Attachment F: Material Control and Accountability Lines of Inquiry ............................................. C-16

Attachment G: Classified Matter Protection and Control Lines of Inquiry ..................................... C-17


January 2017 C-4



(Facility Name)

(DATE)

ASSESSMENT PLAN AND APPROACH

I. INTRODUCTION

The Office of Enterprise Assessments (EA) will conduct a comprehensive safeguards and security

(S&S) assessment at the (insert site/facility), (insert dates). Assessment activities will assess the

effectiveness of management processes and mission operations associated with the protection of

special nuclear material (SNM) and classified matter. The objectives of this assessment are to

evaluate the effectiveness of the (insert site/facility) protection program by examining specific

security topical areas and to provide senior managers within the U.S. Department of Energy (DOE),

or the National Nuclear Security Administration (NNSA), if applicable, with an evaluation of the

adequacy of Departmental policy requirements and their implementation.

EA conducts assessments in accordance with Departmental directives, EA protocols, and

assessment guides. Although this plan outlines projected assessment activities, EA may adjust

specific focus areas, performance testing, and/or documentation requests during the assessment in

response to emerging concerns, requests from senior DOE managers, or performance trends

observed during data collection. If observations indicate a need to deviate substantially from the

documented scope of this plan, such changes will be coordinated with the (insert field office, site,

facility, program office, contractor management, etc., as applicable).

II. APPRAISAL SCOPE

EA will use a variety of performance testing and data collection methods to assess the readiness

of protection systems to counter a broad spectrum of threats and adversary capabilities. To

accomplish the objectives of this assessment, EA will evaluate the following topics and subtopics:

Protection Program Management (PPM); Physical Security Systems (PSS); Protective Force (in

coordination with the Office of Emergency Management, if applicable); Material Control and

Accountability (MC&A); and Classified Matter Protection and Control (CMPC), which will also

include elements of program management and personnel security.

PPM activities will concentrate on the effectiveness and validity of S&S management and

administration; S&S planning and procedures, including security plans, Security Conditions,

performance assurance (i.e., performance of essential elements), surveys, and self-assessment

programs. The S&S training program for non-protective force personnel and the implementation

of the DOE Tactical Doctrine will also be included in this assessment.

PSS assessment activities will include a review of security areas and secure storage facilities,

including the effectiveness of associated intrusion detection and assessment systems, along with

procedures and equipment associated with access control systems and entry/exit inspections. The

comprehensiveness and adequacy of the PSS-related performance assurance program (PAP) and

testing and maintenance activities will be evaluated, including actions associated with false and

nuisance alarms. Assessment activities will incorporate protective force communications, the

lock and key program, lifecycle management of security equipment, and other related topics.


C-5 January 2017

Protective Force assessment activities will focus on protective force management, training,

equipment, facilities, and routine and emergency duty performance. EA will emphasize

performance-based testing, including force-on-force (FOF) exercises that incorporate recapture

and recovery, limited-scope performance tests (LSPTs), firearm proficiency testing, and training

and readiness. EA will evaluate the protective force’s knowledge of and training on Departmental

use-of-force policies and arrest authorities, familiarity with and training on approved rules of

engagement, and the consistency of protective force operations and plans as set out in the site

security plan (SSP) documentation. The integration of protective force and Office of Emergency

Management operations will be assessed through the review of applicable plans and procedures,

personnel interviews, and applicable protective force performance test activities. Areas of

evaluation include event categorization and classification, initial notifications and

communications, and coordinated actions to protect the health and safety of onsite personnel and

the public.

MC&A assessment activities will include a review of MC&A program planning documentation.

Observations, interviews, and LSPTs will determine the adequacy of program administration,

materials accountability, and materials control. Performance testing activities will address

measurements, tamper-indicating devices, inventory and accounting, and front and back

accountability checks of selected items. EA will interview personnel responsible for these

subtopical areas.

CMPC assessment activities will address site-specific procedures and training of personnel,

operations security, technical surveillance countermeasures, and physical protection and control,

which includes accountability of classified matter, aspects of personnel security, unclassified

foreign visits and assignments, the S&S awareness program, the human reliability program (HRP),

and the incidents of security concern (IOSC) program. Areas for performance testing include

transmission, reproduction, destruction, and front and back checks of accountable matter.

At present, EA assessments include an evaluation of the site’s actions to address the insider threat

in each topical area. In general, EA examines the comprehensiveness of the insider analyses

supporting current SSPs, administrative and engineered controls for access control, deterrence

actions that aid in preventing unauthorized access by either U.S. citizens or foreign nationals, and

removal or misuse of site security interests. EA will identify specific concerns related to the

insider threat in the relevant sections of the final report; the overall assessment of this area will

appear in the executive summary of the final report.

This Assessment Plan includes separate attachments containing lines of inquiry for each topic.

III. ASSESSMENT SCHEDULE

The Appraisal Team Chief and his Deputy will conduct a leadership-scoping visit (insert dates).

The protective force team will conduct an onsite walkdown (insert date) and scenario

development (insert date). The EA-22 PPM, PSS, MC&A, protective force, and CMPC teams

will conduct planning and data review at DOE Headquarters in Germantown, Maryland (insert

date). All topic teams will conduct performance testing and onsite data collection activities

(insert dates). EA will conduct additional LSPTs, analyze and validate overall assessment

results, develop a draft report (insert dates), and present a closeout briefing at (insert office, site,

facility) on (insert date). EA will provide a draft report to (insert office, site, facility, and program

stakeholders) management for factual accuracy review before the closeout briefing. After the


January 2017 C-6

closeout briefing, EA will provide the draft report to the site and the program office points of

contact the following week. All parties will have ten calendar days to review the draft report.

Each topic team will develop individual schedules (in coordination with their Federal and

contractor counterparts) for more specific onsite activities to implement this overall plan.

IV. ASSESSMENT TEAM RESPONSIBILITIES AND ASSIGNMENTS

Mr. (insert EA official), Assessment Team Chief, will be the senior DOE official and EA point

of contact managing the conduct of onsite assessment activities. Mr. (insert EA official), Deputy

Team Chief, will support Mr. (insert EA official). The Assessment Team Chief and his staff will

ensure that evaluation activities are consistent with this plan, documented protocols, and approved

procedures. The appropriate team management, listed below, will address any questions

regarding the planned lines of inquiry (see Attachments C through G).

ASSESSMENT TEAM CONTACT INFORMATION

Name – Team Chief Telephone Email Address

Name – Deputy Team Chief Telephone Email Address

Name – PPM Topic Team Leader Telephone Email Address

Name – PSS Topic Team Leader Telephone Email Address

Name – Protective Force Topic Team Leader Telephone Email Address

Name – MC&A Team Leader Telephone Email Address

Name – CMPC Topic Team Leader Telephone Email Address

Professional Conduct and Conflict Resolution

All EA team members and (insert office, site, facility, as applicable) representatives will represent

their respective organizations with the highest standards of professional conduct, and will interact

in a manner defined by reciprocal professional courtesy, mutual respect, and objectivity. These

standards of conduct will further define the professional relationship between EA and (insert office,

site, facility, as applicable) in supporting effective and efficient execution of the assessment

process.

Potential concerns related to professional conduct will be resolved in a manner that adheres to the

following tenets:

Validate the complaint.

Report and discuss the concern with management of both organizations on site.

Make every attempt to resolve the concern at the lowest organizational level.


C-7 January 2017

V. ASSESSMENT PROCESS

The appraisal process includes:

Scoping, planning, and reviewing the data call.

Collecting data and performance testing.

Analyzing data and determining the causal factors for any identified program performance

weaknesses.

Preparing a report, identifying findings and deficiencies, and conducting management

validation briefings.

Scoping, Planning, and Data Call Review

Scoping, planning, and data call review are instrumental in developing topic-specific schedules to

augment this plan, thus helping to ensure that (insert office, site, facility, as applicable)

representatives are informed of ongoing data collection efforts. Supplemental to these activities,

EA will make all representatives aware of any significant changes in proposed activities and focus

areas.

Collecting Data and Performance Testing

Accurate data collection is a central feature of the appraisal process. Methodologies for collecting

data, as part of this assessment, will entail the conduct of document reviews, facility tours,

interviews, observations, LSPTs, and FOF performance testing. Evaluative criteria established for

data collection and performance tests will be consistent with Departmental requirements, approved

procedures, (insert office, site, facility, as applicable) training, and/or DOE National Training

Center (NTC) standards.

Analyzing Data

Careful examination of the collected data highlights topical program trends and provides evidence

of progress or decline in overall protection program performance. A common emphasis during

analysis is the effectiveness of performance feedback and improvement mechanisms, such as

survey programs, the PAP, self-assessment activities, and associated corrective action mechanisms.

Report Preparation, Findings/Deficiencies Identification, and Management Briefings

The assessment report will identify observed site protection program strengths and weaknesses, as

well as document gaps or shortfalls in DOE policy that adversely affect protection program

performance. (Insert office, site, and facility, as applicable) line management, and policy

organizations as necessary, will have the opportunity to verify the factual accuracy of results before

issuance of the final report. For major appraisals, the cognizant Program Secretarial Officer will

have the opportunity to submit a brief (approximately two pages) written management response to

the conclusions and any recommendations included in the final draft report. If such a response is

submitted, EA will include it as an appendix in the final report.

Findings are performance-based and/or systemic deficiencies that warrant a high level of

management attention and that, if left uncorrected, could adversely affect the DOE mission, worker

safety and health, the public, or national security. In these cases, (insert office, site, facility, as

applicable) will develop appropriate corrective action plans and track them to completion in


January 2017 C-8

accordance with DOE Order 226.1B, Implementation of Department of Energy Oversight Policy,

and DOE Order 227.1A, Independent Oversight Program. Deficiencies may serve as the basis for

one or more findings. The applicable program office will determine whether such deficiencies

warrant entry into an issues management system and/or whether they require formal corrective

actions.

A best practice is a technique, process, or program attribute observed during an appraisal that may

merit consideration by other DOE and contractor organizations for implementation because it: (1)

has been demonstrated to substantially improve the safety or security performance of a DOE

operation; (2) represents or contributes to superior performance (beyond compliance); (3) solves a

problem or reduces the risk of a condition or practice that affects multiple DOE sites or programs;

or (4) provides an innovative approach or method to improve effectiveness or efficiency.

Opportunities for improvement are suggestions offered that may assist cognizant managers in

improving programs and operations. These suggestions are not prescriptive.

EA will provide briefings to the appropriate (insert office, site, and facility, as applicable) managers,

highlighting the analyzed results of EA’s data collection efforts to aid in management resource

decisions.


C-9 January 2017

ATTACHMENT A

ASSESSMENT SCHEDULE

Scoping, Data Collection, Performance Testing, Report Writing, and Closeout Activities

Assessment Phase Date Location Activity

Leadership Scoping [insert date] [insert site] Coordinate assessment activities with site

and DOE/NNSA leadership.

FOF Scoping [insert date] [insert site] Coordinate assessment activities, scenario

development scoping, and logistical

support requirements.

EA Headquarters

Planning Meeting

[insert date] Germantown, MD All topic teams review documents and

plan data collection activities.

Scenario

Development

[insert date] [insert site] Protective Force Performance Testing

Scenario development.

Data Collection/

Performance Testing

[insert date] [insert site] All topic teams continue data collection

and performance testing activities.

Data Collection and

Analysis

[insert date] [insert site] All topic teams continue data collection,

performance testing activities; teams

analyze data and assist in the preparation

of the draft assessment report.

Analysis and Report

Writing

[insert date] [insert site] All topic teams analyze data and prepare

draft assessment report.

Closeout Activities [insert date] [insert site] Assessment team leadership conducts

closeout briefing.

Factual Accuracy

Review Comments

[insert date] [insert site] Comments from the program office are

due to EA.


January 2017 C-10

ATTACHMENT B

ASSESSMENT TEAM COMPOSITION

Team Management and Support

Name - Team Chief

Name - Deputy Team Chief

Name - Technical Advisor

Name - Lead Writer

Name - Field Administrative Coordinator(s)

Protection Program Management

Name - Topic Team Lead

Name - Writer

Names of PPM Team Members

Physical Security Systems


Name - Writer

Names of PSS Team Members

Protective Force


Name - Writer

Names of Protective Force Team Members

Material Control and Accountability


Name - Writer

Names of MC&A Team Members

Classified Matter Protection and Control


Name - Writer

Names of CMPC Team Members


C-11 January 2017

ATTACHMENT C

PROTECTION PROGRAM MANAGEMENT

LINES OF INQUIRY

S&S Management and Administration

1) Has the program office and site management provided adequate direction in the areas of Delegations

of Authority, Directives Implementation, and Deviations from Departmental requirements?

2) Has management implemented a personnel development and training program?

S&S Planning

1) Does the site have an SSP/facility security plans, and do they reflect site operations and describe all

aspects of S&S operations?

2) Are security plans supported by a sufficient and documented analytical basis to ensure that protection

requirements can be met?

3) Is the DOE Tactical Doctrine adequately applied?

4) Has an immediately implementable Security Conditions Response Plan been developed?

S&S Program Assurance

1) Are surveys and self-assessments conducted to ensure that S&S systems and processes at facilities/sites

are operating in compliance with Departmental-, national-, and site-level requirements?

2) Do the site’s issues management programs provide the means for timely identification, correction of

deficiencies, and validation of the effectiveness of corrective actions implemented to prevent

recurrence?

3) Has the site established a PAP that identifies the essential elements of the protection program and

establishes monitoring and testing activities of sufficient rigor to ensure that the program elements are

at all times operational and functioning as intended?


January 2017 C-12

ATTACHMENT D

PHYSICAL SECURITY SYSTEMS

LINES OF INQUIRY

Systems Management

1) Does the physical protection system protect Departmental elements from malevolent acts, and are

security areas established?

2) Is there an effective means of assessing alarms?

3) Does management provide adequate resources, including personnel (plus training), equipment, and

facilities to meet the requirements contained in the procedures and policies?

Access Control

1) Are personnel and vehicle access control mechanisms in place and effective?

Vaults and Vault-type rooms

1) Are vaults and vault-type rooms used for open storage of classified mater and/or is SNM effectively

alarmed and controlled?

Locks and Keys

1) Has a lock and key program been established in a graded fashion?

2) Does the program include a use-and-protection strategy for grand master, master, and sub-master, and

have the control keys been evaluated and documented in the SSP?

3) Has an inventory system been implemented to ensure the accountability of Level I, II, and III security

locks, keys, key rings, key ways, and pinned cores?

Testing and Maintenance

1) Are physical protection systems, including components, part of a scheduled testing and maintenance

program?

2) Is screening equipment capable of detecting prohibited and controlled articles before being permitted

into DOE facilities?

Barriers

1) Are barrier delay mechanisms used to deter and delay access to, and removal or unauthorized use of,

Category I/II SNM, and do these mechanisms reduce reliance on recapture/recovery operations?

Communications

1) Do systems remain operable during the loss and recovery of primary electrical power?

2) Is the radio system designed to resist eavesdropping and transmission of deceptive messages and to

protect against jamming?

Closed-Circuit Television and Lighting

1) Does the lighting system allow for detection and assessment of unauthorized persons?

Intrusion Detection Systems

1) When protecting Category I/II SNM, is intrusion detection and assessment immediate?

2) Has the intrusion detection system been designed, installed, operated, and maintained to ensure that the

number of false and nuisance alarms do not reduce system effectiveness and to deter adversaries from

circumventing the detection systems? Have gaps in detection coverage been eliminated?

3) If an early warning intrusion detection system is used to supplement the perimeter intrusion detection

and assessment system, have false and nuisance alarm rates been established that do not degrade the


C-13 January 2017

overall effectiveness of the system, including personnel monitoring the system’s ability to assess and

manage the alarms?

4) Does the intrusion detection system have primary electrical power from onsite power?

5) Can a single person, such as a system administrator or maintenance technician, modify settings to

obscure or prevent alarm signals from being properly identified by the alarm monitoring system

operator?

Entry/Exit Screening

1) Is an inspection program documented in the SSP that ensures prohibited and controlled articles are

detected and DOE assets are not removed when entering/exiting the Protected Area or material access

area?

2) Has the facility analyzed the potential for an adversary to use explosives to affect such consequences

as sabotage or theft? If warranted, have protection measures been identified, approved by the Officially

Designated Security Authority, and documented in the SSP?

Badge Office Operations

1) If local site-specific only badges are used, has DOE line management prescribed or approved

procedures for their design issuance, use, accountability, and return?

2) Does the site maintain records of issued badges, including disposition, description and badge number,

date of issuance, name, organization, and date of destruction and destruction certificate?

3) Are records maintained showing disposition of credentials and shields to include those that are lost? If

lost, is it reported immediately?


January 2017 C-14

ATTACHMENT E

PROTECTIVE FORCE

LINES OF INQUIRY

Management

1) Do directives, plans, and general and special orders meet DOE requirements for currency, clarity, and

applicability to site-specific standards?

2) Do protective force programs, functions, or activities incorporate basic planning principles to ensure

that they accomplish their intended purpose, in accordance with DOE S&S requirements?

3) Is sufficient operational guidance provided through the establishment and maintenance of a formalized

written directives system to ensure that protective force missions/functions are accomplished as

intended?

4) Do protective force personnel comply with the Departmental medical, physical readiness, and firearms

qualifications and training requirements outlined in 10 CFR Part 1046?

5) Are protective force self-assessment and corrective action programs adequately implemented?

6) Are adequate numbers of supervisors assigned to all shifts, and do the supervisors provide the necessary

level of supervision required by local and DOE orders and policies?

Training

1) Is training conducted in accordance with a DOE NTC-approved program in accordance with the

training approval program and provided to ensure performance of assigned functions and tasks under

both normal and emergency conditions?

2) Is the formal training and qualification program based on a valid and complete set of job tasks, with

identified levels of skills and knowledge needed to perform the essential functions outlined in 10 CFR

Part 1046?

3) Are the knowledge, skills, and abilities necessary to competently perform the tasks associated with

assigned protective force duties identified based upon the job analysis/Mission Essential Task List

applicable to each job assignment?

4) Do protective force personnel who are assigned instructor duties have current certification to the level

of training delivered?

5) Have all firearms instructors successfully completed the DOE Firearms Instructor course offered at

NTC (instructors for Security Police Officer IIs and IIIs must complete the Advanced Weapons Systems

Instructor Certification course)?

Equipment and Facilities

1) Is the protective force equipped and provided with the necessary resources to effectively, efficiently,

and safely perform both routine and emergency duties in daylight or under reduced visibility

conditions?

2) Is equipment, specifically weapons and communications systems, tailored to effectively combat and

defeat adversaries identified in the Department’s threat policy (i.e., DOE Order 470.3B, Graded

Security Protection Policy, and DOE Order 470.3C, Design Basis Threat Policy, as applicable) and

site-specific threat guidance or as specified in the SSP under all environmental and tactical conditions?

3) Does the site have an armorer with the knowledge, capability, and responsibility for inspecting,

maintaining, and repairing all firearms available for use?

4) Are permanent (routine and emergency duty) posts that control access constructed consistent with the

vulnerability analysis as documented in the SSP?

5) Are suitable training facilities, to support applicable protective force activities, provided and maintained

based on mission-specific needs?


C-15 January 2017

Duties

1) Do protective force personnel demonstrate familiarity with, and knowledge of, the responsibilities

identified in the job analysis/Mission Essential Task List for their assignment and demonstrate

proficiency in the skills and abilities necessary to perform required and assigned job tasks?

Emergency Response/Emergency Management

1) Has the site developed and implemented an integrated and comprehensive Emergency Management

Plan that identifies protective force requirements?

2) Can the protective force respond effectively and efficiently to all operational emergencies and provide

emergency assistance so that appropriate response measures are taken to protect the worker, the public,

the environment, and national security?

Performance Testing

1) Does the PAP identify the essential elements of the protection program and establish monitoring and

testing activities with sufficient rigor to ensure that the program elements are at all times operational,

functioning as intended, and interacting in such a way as to identify and preclude the occurrence of

adverse activity before security is irreversibly compromised?

2) Does the performance testing program include operability tests to confirm, without any indication of

effectiveness, that a system element or total system is operating as expected?

3) Does the performance testing program include the conduct of effectiveness tests to provide assurance

that essential elements of the system are working as expected, separately or in coordination, to meet

protection program objectives?

Performance Testing Safety

1) Are tests documented in a test report that includes a narrative description of the testing activity and an

analysis of test results?

2) Are issues requiring corrective action documented and tracked until resolved?

3) When unsatisfactory results of a test indicate that national security and/or the health and safety of

facility/site employees or the public is jeopardized, are immediate compensatory measures taken until

the issue is resolved and are normal reporting procedures followed?

4) Does the S&S program incorporate a risk-based approach to protect the health and safety of DOE

Federal and contractor employees or the public?

5) Does the site conduct timely collection, reporting, analysis, and dissemination of information on

environment, safety, and health issues as required by law or regulations or as needed to ensure that

DOE and NNSA are kept fully informed on a timely basis about events that could adversely affect the

health and safety of the public or the workers?

Demonstrator Protestor

1) Has the site developed and implemented plans and procedures to identify demonstrators or protestors

and execute the appropriate response in accordance with DOE policy?

Workplace Violence Active Shooter

1) Has the site developed, in accordance with the DOE Tactical Doctrine, a site-specific plan for managing

site personnel and procedures for managing the disposition of workers in the event of a workplace

violence incident or active shooter attack scenario?


January 2017 C-16

ATTACHMENT F

MATERIAL CONTROL AND ACCOUNTABILITY

LINES OF INQUIRY

Program Management

1) Does site management provide sufficient focus and resources to adequately safeguard nuclear material?

2) Are SNM quantities and categories accurately characterized and appropriate levels of protection

provided?

3) Is the MC&A program adequately documented and approved by DOE?

4) Are personnel trained and qualified to perform MC&A-related activities?

5) Is the effectiveness of the MC&A system periodically verified?

6) Are systems established to maintain accountability of SNM?

7) Are detection systems in place to respond to and report potential losses of SNM?

Material Control

1) Are material balance areas and personnel responsibilities defined, and are internal controls

implemented?

2) Are systems in place to detect the loss or diversion of SNM in a timely manner?

3) Are category limits observed?

4) Is a graded approach in place for SNM use, storage, and protection?

5) Is termination of safeguards properly applied to designated material?

6) Is a comprehensive tamper-indicating device program established and implemented?

7) Do transfers and shipments of SNM include appropriate controls and documentation?

8) Are controls in place to prevent unauthorized removal of SNM in waste streams?

Measurements

1) Is the technical basis for measurements and measurement uncertainty qualified, validated, documented,

and approved by DOE?

2) Are sources of significant measurement uncertainty, including sampling errors, identified and used to

estimate random and systematic errors?

3) Is a statistically-based measurement control system implemented?

4) Are key measurement points established and included in operations procedures?

5) Are uncertainties appropriate and routinely verified to be consistent with approved target values?

6) Are measurement systems, including traceable standards, calibrated and re-calibrated on a defined

frequency?

Material Accounting

1) Are accurate records of nuclear materials maintained, and are transactions and adjustments made in the

system?

2) Are account structures established and maintained?

3) Are the quality, integrity, and capability of the accountability system maintained?

4) Is reporting to Nuclear Materials Management and Safeguards System appropriate?

5) Is a physical inventory regularly conducted?

6) Are physical inventories conducted at defined frequencies?

7) Are sampling approaches used to conduct physical inventories statistically valid?

8) Is the physical inventory process timely and comprehensive?

9) Is the physical inventory based on measured values?

10) Does the site have the capability to conduct emergency physical inventories?

11) Are inventory differences evaluated and investigated?


C-17 January 2017

ATTACHMENT G

CLASSIFIED MATTER PROTECTION AND CONTROL

LINES OF INQUIRY

Classified Matter Protection and Control

1) Has the information security program or CMPC program developed adequate procedures to ensure that

classified information in all forms is protected in accordance with all applicable laws, regulations,

policies, directives, and other requirements?

2) Does the information security program or CMPC program ensure that all individuals with authorized

access to classified information receive instruction with respect to their specific security duties as

necessary to ensure that they are knowledgeable about their responsibilities and applicable?

3) Has the information security program or CMPC program established handling and protection

procedures for classified information throughout its lifecycle?

4) Does the information security program or CMPC program ensure that access to classified information

requires appropriate clearance, relevant access approval, and need to know?

Incidents of Security Concern

1) Has site management established an IOSC process that includes timely identification, notification,

inquiry, reporting, and closure of IOSC?

2) Has a site IOSC program plan been developed that addresses the components of an IOSC program as

outlined in DOE Order 470.4B, Safeguards and Security Program?

3) Does the site’s IOSC program integrate with the site’s PPM function for the purpose of influencing

other programs (i.e., performance assurance, self-assessment, and oversight) and enhancing site-

specific implementation of security policies?

4) Does the site’s issues management programs provide the means for timely identification and correction

of deficiencies in non-compliant conditions to prevent adverse events and validate the effectiveness of

corrective actions implemented to correct identified deficiencies?

Foreign Ownership and Control or Influence (FOCI) and Facility Clearance (FCL)

1) Does the site meet the requirements to possess and secure classified matter or SNM (and, as applicable,

to protect other assets and conduct other security activities on behalf of DOE)?

2) Have the prime contractors implemented provisions pertaining to subcontractors and that all

subcontractors are processed for facility clearances when required and terminated or transferred as

appropriate?

3) Are FOCI and FCL procedures documented to ensure changes to key management personnel are

verified as they occur and that access authorizations are immediately processed for a new key

management personnel?

4) Does the responsible program ensure that the Safeguards and Security Information Management

System database accurately reflects established facilities, security assets, and activities under the local

Federal office’s jurisdiction; ensure that updates and changes to such information are recorded in this

database immediately; and ensure that accurate forms are submitted for this purpose?

Personnel Security, Workplace Substance Abuse, HRP, S&S Awareness, and Unclassified Foreign

Visits and Assignments

1) Does the site’s access authorization process ensure that pre-employment screening, clearance requests,

terminations, and reporting requirements are performed properly and efficiently?

2) Does the site’s workplace drug and alcohol program ensure a drug free workplace?

3) Does the site’s HRP ensure that individuals who occupy designated positions meet the highest standards

of reliability and mental and physical suitability?


January 2017 C-18

4) Does the security awareness program communicate security responsibilities to all individuals and

provide a means to continuously instruct individuals with access authorizations in their duties and

responsibilities?

5) Does the unclassified foreign visits and assignments program provide a process to ensure that

unauthorized access is denied and approved visits and assignments are documented and tracked?

Office of Security Assessments Appendix D Appraisal Process Protocols

D-1 January 2017

Appendix D – Appraisal Scheduling Process


Enterprise Assessments (EA), adheres to strategic planning processes outlined in the Independent Oversight

Program Appraisal Process Protocols. However, EA-22 has developed additional guidance to ensure

adequate coverage and efficiency in the scheduling process, given the complexity of scheduling appraisal

activities across physical and information security programs throughout the Department. This appendix

provides EA-22 staff with detailed expectations and milestones for each phase of the appraisal scheduling

process. The key elements affecting EA-22’s scheduling process are the comprehensiveness of data

considered, site and program office involvement, and transparency.

EA-22 generally conducts appraisal activities every two to three years at sites that protect Category I

quantities of special nuclear material or highly sensitive classified information. To ensure appropriate

prioritization and scheduling of assessments for the sites and facilities with the highest potential risk, EA-

22 uses a formal process for collecting and analyzing a broad range of protection program performance

data, and for coordinating with the applicable program office and site security officials to identify potential

areas of management interest and any potential schedule conflicts.

The analysis process begins approximately six months before each new calendar year starts. This analysis

is based on specific factors (see Appendix E, right column in Table E-1), which EA-22 evaluates using

available empirical data and professional judgment. One product of the analysis is a priority designation

(based on the level of perceived risk) to guide the determination of each site’s priority and how often it will

be subject to scheduled assessment activities. EA-22 uses this information in the context of available time

and resources to develop an assessment schedule for the upcoming calendar year and an initial

determination of the type and scope of each proposed assessment. The assessment schedule includes

proposed dates and estimated resources to support physical security assessments as well as information

security assessments, such as sensitive compartmented information (SCI) facility assessments and classified

matter protection and control (CMPC) special reviews.

After the development of the proposed schedule and resources, EA-22 management convenes internal

meetings with physical and information security subject matter experts and senior EA management to

further evaluate and refine the schedule. Follow-on meetings with the appropriate Headquarters program

offices and site security officials, generally held between June and August of each calendar year, provide

these senior management stakeholders with the proposed EA-22 schedule and an opportunity to

communicate input on desired focus areas or relay feedback on the proposed appraisal dates, type, and

scope. Before discussing plans for limited-notice performance test (LNPT) activities with any personnel

outside of EA, EA-22 requires completion of a trusted agent form to help protect the confidentiality of the

proposed test subject areas and dates.

After the scheduling meetings, EA-22 finalizes the appraisal schedule and formally coordinates with the

program offices and within EA to reconcile any schedule conflicts. The Director, EA-22, then forwards

the final schedule to senior EA management for approval, along with recommendations and the rationale

for any adjustments to the prioritization and/or deferral of assessment activities. After the Director of EA

approves the schedule, EA-22 distributes it to the various DOE program offices, usually by mid-November

of each calendar year.

The milestones and responsible individuals for the actions associated with the phases of the EA-22 appraisal

scheduling process are described below.

Office of Security Assessments Appraisal Process Protocols Appendix D

January 2017 D-2

Milestones and Responsible Individuals

May: Director, EA-22, hosts a meeting at Headquarters to develop an initial schedule and resources:

Receive EA management scheduling guidance.

Review assessment and survey reports.

Review incident reports and enforcement records.

Conduct informal discussions with program offices and site security managers regarding potential

appraisal activities.

Develop an initial schedule of physical/information security assessments and reviews, SCI facility

assessments, and CMPC special reviews, and identify necessary resources.

Brief/provide initial schedule to EA management.

June – August: EA-22 hosts formal scheduling conferences or workshops with the National Nuclear

Security Administration and the DOE Offices of Environmental Management, Science, Nuclear Energy,

and Intelligence and Counterintelligence:

Receive program offices’ input and requests for support.

Ensure that program office personnel sign trusted agent forms where necessary.

Share initial schedule (minus LNPT activities).

Review and analyze recent performance data.

Brief EA management and receive additional guidance.

September: Director, EA-22, integrates and finalizes the appraisal schedule:

Address program office and EA management comments.

October: Director, EA-22, forwards the schedule to EA management for approval.

November: Director, EA-22, transmits the approved annual schedule to program offices and affected field

sites.

Office of Security Assessments Appendix E Appraisal Process Protocols

E-1 January 2017

Appendix E – Factors for Determining a Site’s Appraisal Priority, Type, and Scope

Table E-1. Factors for Determining a Site’s Appraisal Priority, Frequency, Type, and Scope

Priority and Frequency Factors for Determining Appraisal Type and Scope

Priority I:

Sites with high-value assets or with

higher potential risk; includes all sites

that store and use Category I special

nuclear material (SNM) and/or high

consequence classified information

(e.g., Top Secret and sensitive

compartmented information,

Secret/Restricted Data, and/or Sigma

14, 15, 18, and 20).

Under non-emergency conditions,

multi-topic assessments at Priority I

sites are typically conducted every 30

months.

Asset characterization and site mission: The types and

attractiveness of security interests located at a U.S.

Department of Energy (DOE) site and the associated

risks (or perceived risks) are significant considerations in

determining the scope and type of appraisal activity to be

conducted.

Changes in site mission and operations: Significant

changes in site mission/operations, site geography, or the

site security program contract structure are factors that

the Office of Security Assessments (EA-22) evaluates in

the appraisal scoping process.

Changes in policy: DOE policy and other applicable

national standards represent the fundamental measure by

which EA-22 conducts its appraisal activities. Changes

in policy can significantly impact site operations and are,

therefore, important considerations in determining

appraisal scope.

Previous evaluation results: Previous Office of

Enterprise Assessments (EA) assessment results and

regulatory enforcement actions, DOE Inspector General

inspections, site/operations office surveys, and

contractor assurance activities, as well as evaluation

results from external agencies, provide valuable insights

that EA-22 considers during the appraisal scoping

process.

Incidents of security concern: Recent/ongoing major

vulnerabilities, significant incidents, or chronic

recurrence of lesser incidents influence EA-22’s scoping

and prioritization process.

Line management input: EA-22 is committed to support

Headquarters and field line managers in accomplishing

their safeguards and security responsibilities and to assist

them in achieving DOE mission objectives. Therefore,

input regarding requested focus areas from Headquarters

and field line management is an especially important

factor in determining appraisal priority, type, and scope.

Priority II:

Sites with medium-value assets or with

lower potential risk; includes sites with

Category II or III SNM in accessible

and transportable form and/or Secret

(and lesser) classified matter.

Under non-emergency conditions,

multi-topic assessments at Priority II

sites may be conducted every 36 to 48

months.

Priority III:

Sites primarily with non-national

defense missions, or possessing less

than Category III quantities of SNM

and/or very limited classified matter.

Assessment type and frequency at

Priority III sites are typically directed

by EA management on an “as needed”

basis consistent with insights gained in

reviewing the factors for determining

appraisal type and scope.

Office of Security Assessments Appraisal Process Protocols Appendix F

January 2017 F-1

Appendix F – Data Analysis, Deficiencies, Findings, Opportunities for Improvement, Recommendations, Best Practices, and Ratings

Data Analysis


Enterprise Assessments (EA), conducts appraisals to assess the extent to which DOE sites comply with

current program requirements and how well site protection programs are performing. While DOE security

policy requires sites to perform certain functions and achieve certain levels of protection, policy does not

always contain specific information on what measures must be taken or how to achieve an appropriate level

of protection. Therefore, EA-22 appraisal activities emphasize an approach that builds on both compliance

and performance measures to make the most accurate evaluation of the effectiveness of the protection

program.

While analysis is an ongoing process during all phases of an appraisal, it culminates during the reporting

phase. Analysis involves a critical review of all data collection results, particularly any identified program

strengths or deficiencies (inadequacies), and leads to logical, supportable conclusions regarding how well

the program functions and satisfies the intent of DOE and national security policy.

In accordance with DOE Order 227.1A, EA-22 documents appraisal results that clearly identify findings,

opportunities for improvement, and ratings. When applicable, EA-22 has chosen to also document program

shortfalls that do not impact risk and performance (such as the failure to meet a compliance requirement)

as deficiencies. A detailed discussion of deficiencies, findings, and opportunities for improvement is

provided below.

Deficiencies

A deficiency is an inadequacy (e.g., failure to implement a requirement or meet a performance standard)

identified during an appraisal. Deficiencies may serve as the basis for one or more findings.

When the assessment team identifies deficiencies, the analysis must consider these weaknesses individually

and collectively, and the team must then evaluate/analyze them in the context of identified strengths or

mitigating factors to determine their overall impact on the program’s effectiveness. Once a deficiency has

been identified, the team must determine its significance and how it will be documented in the appraisal

report. Factors considered during analysis include:

The magnitude and significance of the actual or potential vulnerability to DOE security interests

resulting from the deficiency.

The deficiency’s actual or potential effect on mission performance.

Site knowledge of the deficiency and corrective actions. For example, findings may be warranted if

management self-identified a weakness but did not implement timely or appropriate corrective actions.

Whether the observed deficiency is isolated or systemic. Systemic deficiencies – those that are

widespread, chronic, or recurring – are more serious.

Other effective programs or program elements that could mitigate the impact of a deficiency.

Figure F-1 at the end of this appendix presents a tool to help EA-22 assessors determine the significance of

a given deficiency.

Office of Security Assessments Appendix F Appraisal Process Protocols

F-2 January 2017

Examples of deficiencies that do not rise to the level of a finding:

(Note: There is no correlation between the example deficiencies; they are separate, standalone examples.)

Scenario: The assessment determined that Federal personnel conduct periodic safeguards and security

surveys and identify appropriate, meaningful findings. Further, the contractor conducts adequate causal

analyses of the findings, develops appropriate milestones for their correction, closely monitors corrective

actions to ensure that they are properly accomplished in a timely manner, and verifies the efficacy of the

corrective actions. Even though the attendant records of these activities are properly entered into a

contractor corrective actions management system, the findings are not being entered into the Safeguards

and Security Information Management System.

Deficiency: The field office has not entered and tracked survey findings in the Safeguards and

Security Information Management System as required. [DOE Order 470.4B, Appendix A, Section 2,

paragraph 9.b]

Scenario: EA’s review of contractor’s training approval program (TAP) certificate indicated that it was

last approved in 2009. Further review determined that the contractor had taken no action to submit an

updated TAP request packet to the National Training Center (NTC) for review and

approval/certification. No other significant deficiencies in training or training performance were noted.

Deficiency: The contractor has not ensured that the TAP certification is current or was submitted

to the NTC for approval. [DOE Order 473.3A, Attachment 2, Section B, paragraph 1.a]

Scenario: Classified matter is stored in approximately 150 General Services Administration-approved

security containers and vault-type rooms throughout the site. EA’s review determined that all repositories

contained the proper documentation, and adequate physical protection for classified matter. Due to a recent

voluntary reduction in force at the site, many repository custodians have terminated employment, requiring

a large number of combination changes. Except for a single contractor organization, most combinations

have been changed in a timely manner. At the time of the assessment, there were six repository access lists

containing names of 12 personnel no longer employed by the contractor. Further review determined that

the combinations for these six repositories had not been changed for over two months. As partial mitigation

of this condition, contractor management had previously recovered the terminated employees badges and

canceled their access privileges in the automated entry control system to help prevent unauthorized access

to the repositories.

Deficiency: The contractor did not change classified repository combinations as soon as practical to

preclude access by individuals who no longer have a need to know. [DOE Order 471.6, Change 2,

paragraph 4.b.(5)(d)4]

Findings

Findings are deficiencies that warrant a high level of attention on the part of management. If left

uncorrected, findings could adversely affect the DOE mission, the environment, worker safety or health,

the public, or national security. Findings define the specific nature of the deficiency and whether it is

localized or indicative of a systemic problem, and identify which organization is responsible for corrective

actions. Consequently, findings provide for a high level of management attention and focus on improving

protection program performance. Team members are responsible for determining which appraisal results

are designated as findings. Minor and non-systemic deficiencies must be appropriately identified so that

they can be corrected, but they are normally not designated as findings.


January 2017 F-3

EA-22 documents findings by discussing the specific problem, identifying all relevant factors, describing

the impact on protection, and referencing the appropriate requirement. Assessment team members are

reminded that with the publication of revised DOE safeguards and security directives and policies, many

of the specific requirements (regarding how to meet program objectives) contained in prior directives were

not carried over into the current directives. Therefore, findings may have to be based on overall program

goals, objectives, and requirements; local implementing procedures; or relevant Federal standards.

If multiple identified deficiencies address specific aspects of a single standard, they should be “rolled up”

and reported as a single finding if the single finding statement can clearly and completely convey the

problems observed. Finding narratives are formulated to express the specific nature of the deficiency,

clearly indicate whether the deficiency is localized or indicative of a systemic problem, and identify which

organization (DOE Headquarters or field element, facility contractor, etc.) is responsible for corrective

actions. In the appraisal report, each finding typically appears after a discussion of the impact of the

condition described, including any mitigating factors and compensatory measures.

Examples of findings:

(Note: As in the examples of deficiencies, these are standalone examples that are unrelated to one another.)

Scenario: This assessment determined that several findings from the last EA appraisal either were not

corrected or had recurred. Such repeat weaknesses included incomplete vulnerability assessment reports

(which continued to omit the same necessary items); continued excessive false and nuisance alarms

prompted by inadequate preventive or corrective maintenance; and the protective force’s inappropriate use

of force. EA found that the contractor does not monitor and track issues to ensure that causal factors,

corrective action milestones, timeliness of actions, or effectiveness reviews are appropriately analyzed so

that issues are corrected in a manner that prevents recurrence. As a result, the organizations continue to

experience systemic problems that impact multiple protection programs.

Finding: The contractor has not implemented timely, effective corrective actions for issues identified

in self-assessments and surveys and has not validated the effectiveness of corrective actions to prevent

recurrence of the issues. [DOE Order 474.B, Attachment 2, Section 2, paragraph 5.f ]

Scenario: This assessment determined that protective force personnel could not demonstrate the requisite

skills in using multiple firearms systems. While on the range, several protective force officers failed to

keep their M-4 rifle muzzles pointed downrange at all times when on the firing line, while others could not

(or did not) take immediate actions to correct rifle malfunctions. Further, only 70 percent of the shooters

were able to qualify with the M-203 40mm grenade launcher on the first attempt. These examples represent

a systemic problem in weapons training that significantly impacts protective force readiness.

Finding: The contractor protective force did not demonstrate the requisite firearms proficiency with

the M-4 rifle and the M-203 40mm grenade launcher. [DOE Order 473.3A, Attachment 2, Section

B, paragraphs 2.a and 5.a]

Scenario: EA determined that the site has developed the required human reliability program (HRP)

implementation plan covering most persons identified in 10 CFR 712, and is implementing the HRP in

accordance with that plan. However, the protective force armorers were not enrolled in the program.

Because these individuals have unescorted access to the weapons that the protective force relies on to

perform its mission, they represent a significant potential insider threat capable of rendering the protective

force ineffective.

Office of Security Assessments Appendix F Appraisal Process Protocols

F-4 January 2017

Finding: The contractor has not ensured that armorers who have unescorted access to firearms are

enrolled in the HRP. [DOE Order 473.3A, Attachment 2, Section A, paragraph 3.b]

Opportunities for Improvement

EA assessors have a broad range of knowledge in their individual topical areas of expertise, and also have

the advantage of observing methods of program implementation across the entire DOE complex. When

they identify deficiencies or inefficiencies in program implementation during an appraisal activity, they can

provide insights on approaches that line managers could adopt to improve program performance. Often,

these suggestions are based on successful approaches observed at other DOE sites. EA-22 identifies such

opportunities for improvement in appraisal reports; however, they are provided only in the context of

recommendations for consideration by line managers, not as directed actions. Opportunities for

improvement that correlate to findings or deficiencies are normally provided as suggested approaches that

line management may consider in their corrective action plans.

Examples of opportunities for improvement:

Protective force management should consider enhancing overall firearms proficiency for Site X

security police officers (SPOs). Specific actions to consider include:

Temporarily increasing the frequency of SPO training and qualifications for the M-4 rifle, 40mm

grenade launcher, and M-249 machinegun until negative performance trends no longer exist.

Developing and implementing practical shooting courses of fire (in a simulated combat course

environment) for the M-4 rifle, 40mm grenade launcher, and M-249 machinegun to enhance SPOs’

proficiency with these firearms and to prepare SPOs to use these firearms more effectively in the

event of an actual adversary attack.

Site information technology managers should consider modifying the vulnerability scanning process

to provide more timely information on actual protection status. Technical actions to consider include:

Conducting network scanning more often than quarterly.

If possible, conducting vulnerability scans monthly as soon as possible after the regular software

patches are applied and virus definitions are updated, to minimize the number of false positives

created by slightly outdated patches or virus definition files.

Recommendations, Best Practices, and Ratings

The EA Independent Oversight Program Appraisal Process Protocols provides guidance on

recommendations, best practices, and ratings.


January 2017 F-5

RD

= R

estr

icte

d D

ata

SA

P =

Sp

ecia

l A

cces

s P

rogra

m

SC

I =

Sen

siti

ve

Com

par

tmen

ted

In

form

atio

n

SN

M =

Sp

ecia

l N

ucl

ear

Mat

eria

l

Fig

ure

F-1

. D

efic

ien

cy S

ign

ific

ance

Det

erm

inat

ion

Pro

cess

Documents

OFFICE OF SECURITY ASSESSMENTS APPRAISAL PROCESS …