DCP_NRC_002912June 8, 2010
ENCLOSURE 4
APP-OCS-GEH-322 Revision D,
"Human Factors Engineering Integrated System Validation Plan"
(Non-Proprietary)
05601jb.doc
Westinghouse
Westinghouse Non-Proprietary Class 3
AP1000TM
Human Factors Engineering Integrated SystemValidation Plan (Non-Proprietary)
APP-OCS-GEH-322,Rev. D
May 2010
APPROVALS
Function Name and Signature
Author Julie I. Reed*Fellow Engineer, Human Factors and Operations
Reviewer Zhonghai Li*Senior Engineer, Human Factors and Operations
Santiago A. Alvarez*Program Manager, Operation and Control Centers
Approver Michael C. Shaffer*Manager, Human System Interface Design
*Electronically approved records are authenticated in the electronic document management system.
© 2010 Westinghouse Electric Company LLCAll Rights Reserved
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
LIST OF CONTRIBUTORS
Revision Name and Title
A Santiago A. AlvarezProject Manager, Human System Interface Design
A Sarah J. BarczykTechnical Editor, Technical Writing and Editing Group
B Todd W. Van MeterSenior Engineer, Simulations and Applications Engineering
B Mark A. CaskeyPrincipal Engineer, Human Factors and Operations
B Megan L. HollandEditorial Specialist, Technical Communications
B, C, D ZhonghaiLiSenior Engineer, Human Factors and Operations
B, C Ruiqi MaPrincipal Engineer, Human Factors and Operations
B Julie I. ReedPrincipal Engineer, Human Factors and Operations
B Zachary T. CasellaProject Engineer, Control Room Design
C, D Daniel ZinzowEngineer, Human Factors and Operations
C Robert B. FuldPrincipal Engineer, Human Factors and Operations
C Mark G. WilliamsSupervisor, AP1000 Operations
C, D Sarah E. GasbarriniTechnical Editor, Technical Communications
D Donald W. BriggsPrincipal Engineer, Human Factors and Operations
Template Version 2.0
APP-OCS-GEH-322, Rev. D i
APO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
REVISION HISTORY
RECORD OF CHANGES
Revision Author Description Completed
A Robert B. Fuld Preliminary Issue 06/08
B Robert B. Fuld Added scenario specifications and other details across 05/09Zachary T. Casella sections. Added proprietary markings.
C Julie I. Reed Tracked changes are not shown in this document due
to extensive revision. These are summarized below:
Deleted Appendices A, E, and F (which are now part
of APP-OCS-GEH-321). Provided cross-references tothis document.
Deleted Appendices B and D. Replaced with the newAppendices A, B, C, and D.
Rev. B of Appendix B is now Appendix E, andAppendix G is now Appendix F.
Updated document to provide more detailedinformation in the following sections:
Section 1.2 to include detailed objectives.
Section 1.3 to address the Technical Support Center.
Section 2.1 to describe the Engineering Development
Facility.
Combined Section 3.2 into Section 3.1 to includeinformation on trial replications, the requirements to
run a fourth trial and the limitations to makingchanges part way through ISV.
Section 4.1 revised to describe the qualifications of
the test subjects.
Section 4.1.2 to describe the minimum and maximum
staffing levels.
Section 4.3 to include test staff training information.
Section 5.2.1 to delete information on procedures that
is now included in APP-OCS-GEH-321.
Section 6.2.1 to include a description of thequestionnaires and observer guides.
Section 6.2.2 to include a description of the situation
awareness measurement technique.
02/10
APP-OCS-GEH-322, Rev. D ii
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
REVISION HISTORY (cont.)
Revision Author Description Completed
C Julie I. Reed Section 7.3 to provide a detailed description of the 02/10(cont.) re-test criteria and to add Figure 7.3-1.
D Julie I. Reed The first issue of this document (APP-OCS-GEH-322) See EDMSis non-proprietary and is being issued at revision D tocorrespond to the revision of the proprietary document(APP-OCS-GEH-320).
Added Section 1.5, "List of Exceptions fromWCAP-15860."
Added a description of the Remote ShutdownWorkstation mock-up to Section 2.1.
Combined Section 3.3 into Section 3.2, "TrialAssignments and Scheduling." Information added onthe assignment of crews to the scenarios.
Expanded Section 4.1.1 to include guidance on theselection of the participant personnel.
Updated Section 5.2, "ISV Procedures" to includeadditional information.
Completely revised Section 6, "Data."
Added information to Section 7.3 to describe therationale for re-running a trial in the event of a smallnumber of Priority 1 and/or Priority 2 human errordiscrepancies (HEDs) based on diagnostic criteria.
Added proprietary markings.
DOCUMENT TRACEABILITY & COMPLIANCE
Created to Support the Following Document(s) Document Number Revision
Programmatic Level Description of the AP1000 Human WCAP-15860 2Factors Verification and Validation Plan
OPEN ITEMS
Item Description Status
None.
APP-OCS-GEH-322, Rev. D i°°
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
TABLE OF CONTENTS
Section Title Page
LIST O F CON TRIBU TO R S ....................................................................................... i
REV ISIO N H ISTO RY ................................................................................................ ii
TA BLE O F CO N TEN TS ............................................................................................. iv
LIST O F TA BLES ....................................................................................................... Vi
LIST O F FIG U RES ..................................................................................................... Vi
A CRO N YM S A N D TRA D EM A RK S ......................................................................... Vii
G LO SSA RY O F TERM S ............................................................................................ Viii
REFEREN CES ............................................................................................................ ix
B IBLIO G RAPH Y ........................................................................................................ xi
SECTIO N I IN TRO D U CTIO N ....................................................................................................... 1-1
1.1 BA CK G RO UND ......................................................................................................... 1-1
1.2 PURPO SE .................................................................................................................... 1-2
1.3 SCO PE ......................................................................................................................... 1-3
1.4 SCO PE O F ISV ISSU ES ............................................................................................. 1-4
1.5 LIST O F EX CEPTIO N S FRO M W CA P-15860 ......................................................... 1-4
SECTIO N 2 TH E ISV FA CILITY ................................................................................................... 2-1
2. i PH Y SICA L SCO PE A N D FID ELITY ........................................................................ 2-1
2.2 FUN CTION A L SCO PE A N D FID ELITY ................................................................... 2-3
2.3 SIM U LA TO R TESTIN G ............................................................................................ 2-3
SECTIO N 3 TEST D ESIG N ............................................................................................................ 3-1
3.1 N U M BER O F TRIA L REPLICA TION S .................................................................... 3-1
3.2 TRIAL ASSIGNMENTS AND SCHEDULING ......................................................... 3-2
3.3 PILO T TESTIN G ........................................................................................................ 3-5
SECTION 4 PA RTIC IPA N TS ......................................................................................................... 4-1
4.1 SU BJECTS .................................................................................................................. 4-1
4.1.1 Selection ....................................................................................................................... 4-2
4.1.2 Crew Size and N um ber ................................................................................................ 4-3
4.1.3 Training ........................................................................................................................ 4-4
4.2 O B SERV ERS .............................................................................................................. 4-4
4.3 ISV STA FF .................................................................................................................. 4-5
,APP-OCS-GEH-322, Rev. D iv
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
TABLE OF CONTENTS (cont.)
Section Title Page
SECTION 5 SCENARIO SET ......................................................................................................... 5-1
f5.1 SCENARIO REQUIREM ENTS .................................................................................. 5-1
5 .1.1 E v en ts .......................................................................................................................... 5 -1
5.1.2 P ro cedures .................................................................................................................... 5-3
5.1.3 Complications ......................................... ..................................................................... 5-4
5.2 ISV PROCEDURES .................................................................................................... 5-6
5.2.1 General Procedure and Documentation ....................................................................... 5-6
5 * 2.2 Communications with ISV Personnel .......................................................................... 5-7
5.2.3 Unforeseen Events ....................................................................................................... 5-8
5.2.4 Storage of Data ............................................................................................................ 5-9
SECTION 6 DATA .......................................................................................................................... 6-1
6.1 M EASURES ................................................................................................................ 6-1
6.2 M ETHODS .................................................................................................................. 6-2
6.3 CRITERIA ................................................................................................................... 6-6
6.3.1 Pass/Fail Criteria .......................................................................................................... 677
6.3.2 Diagnostic Criteria ........................................................................................................ 6-8
SECTION 7 PROCESSING OF RESULTS ..................................................................................... 7-1
7.1 RAW DATA PROCESSING ....................................................................................... 7-1
7.2 ANALYSIS AND INTERPRETATION ..................................................................... 7-1
7.3 ADDRESSING HEDS AND RE-TEST REQUIREMENTS ...................................... 7-2
7.4 RESULTS REPORT .................................................................................................... 7-5
APPENDIX A POST-TRIAL QUESTIONNAIRE FOR SUBJECTS ................................................. A-I
APPENDIX B FINAL QUESTIONNAIRE FOR SUBJECTS ............................................................ B-I
APPENDIX C POST-TRIAL QUESTIONNAIRE FOR OBSERVERSý ............................................ C-1
APPENDIX D FINAL QUESTIONNAIRE FOR OBSERVERS ........................................................ D-1
APPENDIX E TRIAL PROTOCOL .................................................................................................... E-I
APPENDIX F DEBRIEFING PROTOCOL ........................................................................................ F-I
APP-OCS-GEH-322, Rev. D
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
TABLE OF CONTENTS (cont.)
LIST OF TABLES
Table Title Page
[
]ac ......................................................... 3-4]a,, ................................................................................................... 6-7
]", ° .............................a................................................................... 6 -8
LIST OF FIGURES
Figure Title Page
[[
ace 1...................................1-]ac .......................................... 7-4
APP-OCS-GEH-322, Rev. D viAPP-OCS-GEH-322, Rev. D vi
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
ACRONYMS AND TRADEMARKS
Acronyms used in the document are defined in WNA-PS-00016-GEN, "Standard Acronyms and
Definitions" (Reference 1), or included below to ensure unambiguous understanding of their use within
this document.
Acronyms Definition
ADS Automatic Depressurization SystemAPS Alarm Presentation System
CPS Computerized Procedure SystemDAS Diverse Actuation System
EDS Engineering Development SimulatorHED Human Error DiscrepancyHEP Human Error ProbabilityISV Integrated System ValidationLBLOCA Large-break Loss of Coolant AccidentMTIS Maintenance, Test, Inspections and SurveillanceNASA National Aeronautics and Space AdministrationPDSP Primary Dedicated Safety PanelsPMS Protection and Safety Monitoring SystemQ&A Question and Answer
SART Situation Awareness Rating TechniqueSBLOCA Small-break Loss of Coolant Accident
SDSP Secondary Dedicated Safety PanelsTDS Training Development SimulatorTLX Task Load Index
API OOOTM is a trademark of Westinghouse Electric Company LLC.
MoraeTM is a trademark of TechSmith Corporation.
All other product and corporate names used in this document may be trademarks or registered trademarks
of other companies, and are used only for explanation and to the owners' benefit, without intent toinfringe.
APP-OCS-GEH-322, Rev. D viiAPP-OCS-GEH-322, Rev. D vii
APO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
GLOSSARY OF TERMS
Standard terms used in the document are defined in WNA-PS-00016-GEN, "Standard Acronyms andDefinitions" (Reference 1), or included below to ensure unambiguous understanding of their use withinthis document.
Term Definitions
Exception A justified (i.e., documented and approved) departure from specifiedguidance or requirements.
A departure of the APIO00TM Human Factors Engineering (HFE) designfrom guidance and criteria identified during the execution of HFEverification and validation activities.
Human EngineeringDiscrepancy
APP-OCS-GEII-322, Rev. D viiiAPP-OCS-GEH-322, Rev. D Viii
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
REFERENCES
Following is a list of references used throughout this document.
1. WNA-PS-00016-GEN (Proprietary), Rev. 4, "Standard Acronyms and Definitions," WestinghouseElectric Company LLC.
2. APP-OCS-GBH-001 (Proprietary), Rev. 1, "AP1000 Human Factors Engineering Program Plan,"Westinghouse Electric Company LLC.
3. WCAP-15860, Rev. 2, "Programmatic Level Description of the AP1000 Human FactorsVerification and Validation Plan," Westinghouse Electric Company LLC.
4. WCAP-14655, Rev. 1, "Designer's Input for the Training of the Human Factors EngineeringVerification and Validation Personnel," Westinghouse Electric Corporation.
5. NUREG-071 1, Rev. 2, "Human Factors Engineering Program Review Model," U.S. NuclearRegulatory Commission, February 2004.
6. APP-OCS-GEH-321 (Proprietary), Rev. B, "AP1000 Human Factors Engineering IntegratedSystem Validation Scenario Information," Westinghouse Electric Company LLC.
7. APP-OCS-GEH-120 (Proprietary), Rev. B, "APIOO Human Factors Engineering DesignVerification Plan," Westinghouse Electric Company LLC.
8. APP-OCS-GGR-1 10 (Proprietary), Rev. 1, "AP1000 Technical Support Center and Emergency
Operations Facility Workshop," Westinghouse Electric Company LLC.
9. ANSI/ANS-3.5-1998, "Nuclear Power Plant Simulators for Use in Operator Training andExamination," American National Standards Institute/American Nuclear Society.
10. WCAP-16096-NP-A, Rev. I, "Software Program Manual for Common Q Systems," WestinghouseElectric Company LLC.
11. WNA-SQ-00047-GEN (Proprietary), Rev. 0, Standard Integrated I&C Validation Test Strategy,"Westinghouse Electric Company LLC.
12. APP-OCS-GEH-420 (Proprietary), Rev. B, "AP1000 Human Engineering Discrepancy ResolutionProcess," Westinghouse Electric Company LLC.
13. APP-OCS-GEH-220 (Proprietary), Rev. B, "AP1000 Human Factors Engineering Task Support
Verification Plan," Westinghouse Electric Company LLC.
14. APP-OCS-GJR-003 (Proprietary), Rev. 1, "AP1000 Main Control Room Staffing Roles andResponsibilities," Westinghouse Electric Company LLC.
APP-OCS-GEH-322, Rev. D ix
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
REFERENCES (cont.)
15. APP-GW-GL-01 I (WCAP-16555), Rev. 0, "AP1000 Identification of Critical Human Actions and
Risk Important Tasks Form 36 AP1000 NRC," Westinghouse Electric Company LLC.
16. APP-GW-GLR-040, Rev. 1, "Plant Operations, Surveillance, and Maintenance Procedures,"Westinghouse Electric Company LLC.
17. NUREG-1021, Rev. 9, "Operator Licensing Examination Standards For Power Reactors,"
U.S. Nuclear Regulatory Commission, July 2004.
18. Taylor, R. M., "Situational awareness rating technique (SART): The Development of a Tool for
Aircrew Systems Design." In "Situational Awareness in Aerospace Operations
(AGARD-CP-478)," Neuilly-sur-Seine, France: NATO-AGARD, 3/1-3/17, 1990.
19. Taylor, R. M., and S. J. Selcon, "Situation in Mind: Theory, Application and Measurement of
Situational Awareness." In R. D. Gilson, D. J. Garland, & J. M. Koonce (Eds.), "SituationalAwareness in Complex Settings," Daytona Beach, FL: Embry-Riddle Aeronautical University
Press, 69-78, 1994.
20. APP-GW-GAP-100 (Proprietary), Rev. 20, "Inter-Business Unit Policies and Procedures Manual,"
Westinghouse Electric Company LLC.
21. APP-OCS-GEH-520 (Proprietary), Rev. B, "AP1000 Plant Startup Human Factors EngineeringDesign Verification Plan," Westinghouse Electric Company LLC.
22. APP-OCS-JIR-220 (Proprietary), Rev. B, "Operational Sequence Analysis (OSA-2) SummaryReport," Westinghouse Electric Company LLC.
23. APP-OCS-GJR-001 (Proprietary), Rev. 0, "Human Factors Engineering Operating Experience
Review Report for the AP1000 Nuclear Power Plant," Westinghouse Electric Company LLC.
APP-OCS-GEH-322, Rev. D x
APP-OCS-GEH-322, Rev. D X
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
BIBLIOGRAPHY
Following is a list of sources that were considered in preparing this document, or that provide additional
information.
1. APP-GW-GL-700 (Proprietary), Rev. 17, "AP 1000 Design Control Document," Westinghouse
Electric Company LLC.
(Last Page of Front Matter)
APP-OCS-GEH-322, Rev. D xi
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
SECTION 1INTRODUCTION
1.1 BACKGROUND
The Integrated System Validation (ISV) provides a comprehensive human performance-based assessmentof the final design of the API000TM Human-System Interface (HSI) resources, based on their realisticoperation within a simulator-driven Main Control Room (MCR). [
]a,c,e
a,c,e
L ace
a,c,e
APP-OCS-GEH-322, Rev. D 1-1APP-OCS-GEH-322, Rev. D 1-1
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c,e
1.2 PURPOSE
This document describes the ISV implementation plan for AP 1000 as directed by APP-OCS-GBH-00 1,"Human Factors Engineering Program Plan" (Reference 2), WCAP-15860, "Programmatic LevelDescription of the AP 1000 Human Factors Verification and Validation Plan" (Reference 3),WCAP-14655, "Designer's Input for the Training of the Human Factors Engineering Verification andValidation Personnel" (Reference 4), and NUREG-07 11, "Human Factors Engineering Program ReviewModel" (Reference 5).
]ac
APP-OCS-GEH-322, Rev. D 1-2
APO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
axc
The main body of this document includes the scope, methodology, description of the ISV facility, testdesign, participants, scenario selection process, test criteria, and the processing and documentation of theresults. The Appendices comprise details on the workload rating forms, post-trial questionnaires, and thedebriefing protocol.
A 'sister' document APP-OCS-GEH-321, "AP1000 Human Factors Engineering Integrated SystemValidation Scenario Information" (Reference 6) provides the detailed information on the scenariodescriptions, scenario-specific objectives, scenario observer guides, plus detailed information on thescope of simulation and simulation testing. This information was placed in a separate document fortwo main reasons. Firstly, this will assist in restricting access to the scenario information from the ISVparticipants, and secondly, it is envisioned that details of the scenario information are likely to be updatedand further developed prior to the implementation of ISV (in comparison, it is anticipated that theinformation contained in this document will not be subject to change).
1.3 SCOPE
The following are utilized to represent the final MCR and HSI design:
* HSI hardware (consoles, visual display units, workstations, keyboards, mice, trackballs, panels,printers, tables, bookcases, etc.)
* HSI software (non-safety control system, safety control system, alarms, displays, soft controls,computerized procedures, local area network displays, etc.)
* Communications facilities
* Plant emergency, operations, surveillance, and maintenance procedures
* Realistic work environment, including the room dimensions and general arrangement
* Operating crews
]ac
APP-OCS-GEH-322, Rev. D 1-3
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
1.4 SCOPE OF ISV ISSUES
WCAP-15860, "Programmatic Level Description of the AP1000 Human Factors Verification and
Validation Plan," (Reference 3) Section 4.1 lists the issues to be addressed by the ISV implementation
plan. These issues are listed below and mapped to the sections of the present document that addresses
them:
" Objectives - Section 1.1, "Background"
* Personnel performance issues - Section 4, "Participants"
* Test methodology and procedures - Section 3, "Test Design"
* Test participants - Section 4, "Participants"
" Test conditions (including plant conditions, operating sequences, and accident scenarios) -
Section 5, "Scenario Test Set"
* HSI description - Section 2, "ISV Facility"
* Performance measures - Section 6, "Data"
* Data analysis - Section 6, "Data"
* Acceptance criteria - Section 6, "Data"
* Processing of results - Section 7, "Processing of Results"
1.5 LIST OF EXCEPTIONS FROM WCAP-15860
a,c
APP-OCS-GEH-322, Rev. D 1-4
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
(Last Page of Section 1)
APP-OCS-GEH-322, Rev. D 1-5
APO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
SECTION 2THE ISV FACILITY
The ISV will be performed at a dedicated, purpose-built facility. The facility will employ a high fidelity,near full-scope simulator to represent the AP1000 systems and the MCR. This simulator will satisfy therequirements of Sections 3 and 4 of ANSIIANS-3.5-1998, "Nuclear Power Plant Simulators for Use inOperator Training and Examination" (Reference 9). The ISV facility will be provided at the EDS basedat the Westinghouse Headquarters Building in Cranberry, Pennsylvania, USA.
2.1 PHYSICAL SCOPE AND FIDELITY
To the extent practical, the HSI resources provided by the EDS will be identical to those to be deliveredfor the actual API000 MCR. These resources will include full hardware representation for the following:
ac
APP-OCS-GEH-322, Rev. D 2-1
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
APP-OCS-GEH-322, Rev. D 2-2
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
2.2 FUNCTIONAL SCOPE AND FIDELITY
The simulation capability of the EDS will be adequate for the procedures, scenarios, and HSI to beexercised in the ISV. Sufficient simulation scope and systems modeling is determined by the scenariosselected for ISV (see Section 5); Features of the simulation that are not relevant to ISV may be of lesserfidelity or omitted. The development of the required simulator models and software are coordinated withthe Simulator Design and Development group. Details of the scope of simulation by individual plantsystems can be found in APP-OCS-GEH-321, Appendix D (Reference 6).
2.3 SIMULATOR TESTING
The EDS and simulator readiness is demonstrated by the following test activities (also see Section 3.4 forfurther information):[
(Last Page of Section 2)
APP-OCS-GEH-322, Rev. D 2-3
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
SECTION 3TEST DESIGN
This section addresses the number of individual trials, requirements for trial replications, and theirassignment to crews. The minimum number of trials is a function of the number of scenarios and thenumber of replications of each scenario.
3.1 NUMBER OF TRIAL REPLICATIONS
]axC
APP-OCS-GEH-322, Rev. D 3-1
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
a,c
3.2 TRIAL ASSIGNMENTS AND SCHEDULING
]a,c
APP-OCS-GEH-322, Rev. D 3-2
AP1O000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]a,c
APP-OCS-GEH-322, Rev. D 3-3
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]aa)C
ca,caC al
APP-OCS-GEH-322, Rev. D 3-4
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
a,c
II
r aC
3.3 PILOT TESTING
The objective of simulator pilot testing in preparation for ISV is to demonstrate that the simulator
responds in a manner similar to the reference unit while utilizing the plant operating procedures.
] ac
APP-OCS-GEH-322, Rev. D 3-5
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
(Last Page of Section 3)
APP-OCS-GEH-322, Rev. D 3-6
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
SECTION 4PARTICIPANTS
This section describes the personnel involved in ISV. The ISV participants include the test subjects,observers, and the EDS staff.
4.1 SUBJECTS
]a,c
APP-OCS-GEH-322, Rev. D 4-1
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I]ac
4.1.1 Selection
[
]ac
APP-OCS-GEH-322, Rev. D 4-2
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
a,c
4.1.2 Crew Size and Number
[
]a,c
APP-OCS-GEH-322, Rev. D 4-3APP-OCS-GEH-322, Rev. D 4-3
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]a1,
4.1.3 Training
[
I a~c
4.2 OBSERVERS
Project-independent ISV observers will be assigned to evaluate performance and generate results data ineach trial. These observers will include one of each of the following specialists:
[
APP-OCS-GEH-322, Rev. D 4-4
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
4.3 ISV STAFF
For each trial, the ISV staff will include at least one of each of the following specialists:
[
]a,c
(Last Page of Section 4)
4-5APP-OCS-GEH-322, Rev. D
AP1O000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
SECTION 5SCENARIO SET
A scenario comprises the simulated events, anticipated procedure usage, and added complications. A setof scenarios has been developed for ISV. Each scenario will be performed in multiple (i.e., replicate)trials. This section describes the selection and development of the representative set of validationscenarios for ISV. Full details of the scenarios can be found in APP-OCS-GEH-321 (Reference 6),Appendix A.
The ISV scenarios are'developed by a multi-disciplinary team. This team includes human factorsspecialists, procedure writers, operator training developers, and personnel from the simulation group. Anumber of the team members possess previous operating experience, thereby also contributing to theactual credibility and realism of the ISV scenarios.
5.1 SCENARIO REQUIREMENTS-
]a,c
5.1.1 Events
Events are identified for ISV in WCAP-15860, "Programmatic Level Description of the API000 HumanFactors Verification and Validation Plan" (Reference 3) Section 4.6 ("Criteria for Selection of Test
Scenarios for Dynamic Evaluations").
]a,c
APP-OCS-GEH-322, Rev. D 5-1
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
APP-OCS-GEH-322, Rev. D 5-2
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
a,c
5.1.2 Procedures
The procedures to be addressed by ISV are outlined in a number of sections in WCAP-15860,"Programmatic Level Description of the APIOOO Human Factors Verification and Validation Plan"(Reference 3).
[
APP-OCS-GEH-322, Rev. D 5-3
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
5.1.3 Complications
[
]a,c
APP-OCS-GEH-322, Rev. D 5-4
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
II
]a~c
APP-OCS-GEH-322, Rev. II 5-5
APP-OCS-GEH-322, Rev. D 5-5
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]ac
5.2 ISV PROCEDURES
5.2.1 General Procedure and Documentation
The ISV procedures for each scenario (i.e., the "scenario package") will be prepared prior to ISV andmaintained under the ISV Coordinator's control in order to prevent the ISV participants from obtainingprior knowledge of the scenarios. The scenario-packages include the following:
a,c
APP-OCS-GEH-322, Rev. D 5-6
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
5.2.2 Communications with ISV Personnel
I
]ac
APP-OCS-GEH-322, Rev. D 5-7
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I .
]a,c
Unforeseen Events5.2.3
[
]ac
APP-OCS-GEH-322, Rev. D 5-8
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I]a,c
52.4 Storage of Data
The ISV Coordinator will be responsible for the ISV documentation and data. In addition to maintainingand controlling the 'scenario packages,' the ISV Coordinator will collect the ISV results data and will bein charge of ensuring that this data is not 'lost.'
]a.c
(Last Page of Section 5)
APP-OCS-GEH-322, Rev. D 5-9
APO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
SECTION 6DATA
The techniques used to collect the ISV results data are described in this section. This data serves as thebasis for determining whether the task goals and performance requirements are achieved, perWCAP-15860, "Programmatic Level Description of the AP1000 Human Factors Verification andValidation Plan" (Reference 3) Section 4.8.
6.1 MEASURES
A set of performance measures are identified and selected to collect data on operator performance, asfollows:
a,c
APP-OCS-GEH-322, Rev. D 6-1APP-OCS-GEH-322, Rev. D 6-1
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
pac
6.2 METHODS
Data collection will use a variety of paper-and-pencil techniques, structured discussions, and digitalrecording methods, as follows:
a,c,e
APP-OCS-GEH-322, Rev. D 6-2
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
II
]ace
APP-OCS-GEH-322, Rev. D 6-3APP-OCS-GEH-322, Rev. D 6-3
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
] a,c e
APP-OCS-GEII-322, Rev. D 6-4APP-OCS-GEH-322, Rev. D 6-4
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
a,c,e
APP-OCS-GEH-322, Rev. D 6-5
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a~c,e
6.3 CRITERIA
This subsection describes how the measurement results are applied to the determination of the success ofthe ISV trials in respect to the pass/fail criteria and the diagnostic criteria. A set of performancemeasures, as described in Section 6.1, will be used which includes measures of the performance of the
plant and personnel.
APP-OS-GEI-322 Rev.D 6-APP-OCS-GEH-322, Rev. D 6-6
APIOOO Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
6.3.1 Pass/Fail Criteria
]ace
I a,c,e a,c,e
]a,c,e
APP-OCS-GEH-322, Rev. D 6-7
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
6.3.2 Diagnostic Criteria
a~c,e
a,c,e
APP-OCS-GEH-322, Rev. D 6-8APP-OCS-GEH-322, Rev. D 6-8
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
a,c,e
APP-OCS-GEH-322, Rev. D 6-9
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
a,c,e
(Last Page of Section 6)
APP-OCS-GEH-322, Rev. D 6-10APP-OCS-GEH-322, Rev. D 6-10
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
SECTION 7PROCESSING OF RESULTS
The final results of the ISV trials will be processed to promptly to determine the overall results andconclusions, identify HEDs and to assign HED priorities, and to assess the need for added trialreplications.
7.1 RAW DATA PROCESSING
[
Ia~c
7.2 ANALYSIS AND INTERPRETATION
II
pc,
APP-OCS-GEH-322, Rev. D 7-1
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
II
I a~c
7.3 ADDRESSING HEDS AND RE-TEST REQUIREMENTS
I
]ac
APP-OCS-GEH-322, Rev. D 7-2. PP-OCS-GEH-322, Rev. D 7-2
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]ac
APP-OCS-GEH-322, Rev. D 7-3APP-OCS-GEH-322, Rev. D 7-3
APIOOO Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)a,c
2,C
ac
APP-OCS-GEII-322, Rev. D 7-4APP-OCS-GEH-322, Rev. D 7-4
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
a,c
7.4 RESULTS REPORT
The final ISV report will document the results and detailed findings from the analysis, including anyHEDs. Any limitations of the ISV plan, implementation, and execution will also be addressed. The basisfor concluding that the AP1000 MCR, HSI resources, procedures, and operator training are adequate (ornot) will be described (i.e., that the integrated system performed acceptably during testing and can beexpected to support safe operation in actual use).
The final results report will be communicated to the Training Group, Procedures Group, and thePRA Group. This will enable the results of ISV to be incorporated into the development/updates of thetraining programs, revisions to procedures, and any updates to the PRA in terms of the assumptionsclaimed on operator performance.
(Last Page of Section 7)
APP-OCS-GEH-322, Rev. D 7-5
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
APPENDIX APOST-TRIAL QUESTIONNAIRE FOR SUBJECTS
A.1 PROTOCOL
[
APP-OCS-GEH-322, Rev. D A-1
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
I axc
APP-OCS-GEH-322, Rev. B A-2APP-OCS-GEH-322, Rev. D A-2
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
A.2 POST-TRIAL QUESTIONNAIRE FOR SUBJECTS
ac
APP-OCS-GEII-322, Rev. D A-3APP-OCS-GEH-322, Rev. D A-3
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]ac
APP-OCS-GEH-322, Rev. D A-4APP-OCS-GEH-322, Rev. D A-4
APIOOO Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
APP-OCS-GEH-322, Rev. D A-5APP-OCS-GEH-322, Rev. Dj A-5
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
APP-OCS-GEH-322, Rev. D A-6
AP1O0O Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
APP-OCS-GEII-322, Rev. D A-7APP-OCS-GEH-322, Rev. D A-7
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]a,c
APP-OCS-GEH-322, Rev. D A-8
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
] a,c
APP-OCS-GEH-322, Rev. D A-9APP-OCS-GEH-322, Rev. D A-9
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]ac
APP-OCS-GEH-322, Rev. D A-10
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]ac
APP-OCS-GEH-322, Rev. D A-ilAPP-OCS-GEH-322, Rev. D A-11
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]ac
APP-OCS-GEII-322, Rev. D A-12APP-OCS-GEH-322, Rev. D A-12
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
I a~c
APP-OCS-GEH-322, Rev. D A-13APP-OCS-GEH-322, Rev. D A-13
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
I a~c
APP-OCS-GEH-322, Rev. D A-14APP-OCS-GEH-322, Rev. D A-14
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]a,c
APP-OCS-GEH-322, Rev. D A-15APP-OCS-GEH-322, Rev. D A-15
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
a~c
(Last Page of Appendix A)
APP-OCS-GEH-322, Rev. D A-16o
APIO00. Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
APPENDIX BFINAL QUESTIONNAIRE FOR SUBJECTS
Note, additional line spaces for the participants to provide comments have been removed in this report.However, the printed version used in ISV will contain adequate space for the participants to providewritten comments.
Name: Position: Crew: Date:
Circle your level of agreement with the statements below based on your experience in all of the precedingtrials.
a,c
APP-OCS-GEH-322, Rev. D B-1
APIOO0 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
APP-OCS-GEII-322, Rev. D B-2APP-OCS-GEH-322, Rev. D B-2
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]ac
APP-OCS-GEH-322, Rev. D B-3
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]ac
APP-OCS-GEH-322, Rev. D B-4
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
II
ja,c
APP-OCS-GEH-322, Rev. D B-5
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
] a,c
APP-OCS-GEH-322, Rev. D B-6APP-OCS-GEH-322, Rev. D B-6
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
II
APP-OCS-GEH-322, Rev. D B-7
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
APP-OCS-GEH-322, Rev. D B-8
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)APlODO Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
II
]ac
APP-OCS-GEH-322, Rev. B B-9APP-OCS-GEH-322, Rev. D B-9
AP1O000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
(Last Page of Appendix B)
APP-OCS-GEH-322, Rev. D B-10
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
APPENDIX CPOST-TRIAL QUESTIONNAIRE FOR OBSERVERS
Note, additional line spaces for the participants to provide comments have been removed in this report.However, the printed version used in ISV will contain adequate space for the participants to providewritten comments.
Name: Scenario:
Trial: Date:
a~c
APP-OCS-GEH-322, Rev. D C-1
AP1O000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]ac
APP-OCS-GEH-322, Rev. B C-2APP-OCS-GEH-322, Rev. D C-2
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]ac
APP-OCS-~EH-322, Rev. D C-3APP-OCS-,GEH-322, Rev. D C-3
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]ac
APP-OCS-GEH-322, Rev. D C-4
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]aoc
APP-OCS-GEH-322, Rev. D C-5APP-OCS-GEH-322, Rev. D C-5
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]ac
APP-OCS-GEH-322, Rev. D C-6APP-OCS-GEH-322, Rev. D C-6
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
ac
APP-OCS-GEH-322, Rev. D C-7
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
II
]ao
APP-OCS-GEH-322, Rev. B C-8APP-OCS-GEH-322, Rev. D C-8
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]ac
APP-OCS-GEH-322, Rev. D C-9APP-OCS-GEH-322, Rev. D C-9
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
(Last Page of Appendix C)
APP-OCS-GEH-322, Rev. D C-10
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
APPENDIX D IFINAL QUESTIONNAIRE FOR OBSERVERS
Note, additional line spaces for the participants to provide comments have been removed in this report.However, the printed version used in ISV will contain adequate space for the participants to providewritten comments.
Name: Date:
Circle your level of agreement with the statements below based on your experience in all of the precedingtrials.
]a,c
APP-OCS-GEH-322, Rev. D D-1APP-OCS-GEH-322, Rev. D D-1
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]a,c
APP-OCS-GEH-322, Rev. D D-2
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]a,c
APP-OCS-GEII-322, Rev. D D-3APP-OCS-GEH-322, Rev. D D-3
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]axc
APP-OCS-GEH-322, Rev. D D-4APP-OCS-GEH-322, Rev. D D-4
APIOOO Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
APP-OCS-GEH-322, Rev. D D-5
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
Ia,c
APP-OCS-GEH-322, Rev. D D-6
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
APP-OCS-GEH-322, Rev. D D-7
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
] a,c
APP-OCS-GEH-322, Rev. D D-8
APIO000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]ac
APP-OCS-GEII-322, Rev. D D-9APP-OCS-GEH-322, Rev. D D-9
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
I
]aoc
APP-OCS-GEH-322, Rev. D D-10
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
II/
]a,c
(Last Page of Appendix D)
APP-OCS-GEH-322, Rev. D D-11
AP1000 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
APPENDIX ETRIAL PROTOCOL
This Appendix provides the protocol and general sequence of events for running the ISV trials.
PRETRIAL PREPARATIONS
]a,c
START TRIAL RUNII
]a,c
APP-OCS-GEH-322, Rev. D E-lAPP-OCS-GEH-322, Rev. D E-1
AP1000
END TRIAL RUN
I
Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,cBREAK
(Last Page of Appendix E)
E-2APP-OCS-GEH-322, Rev. D
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
APPENDIX FDEBRIEFING PROTOCOL
An informal debriefing of the test participants will be performed after each separate trial run. A formaldebriefing will be held after each major period (typically one week) of testing. Guidance for the formaldebriefing process is provided as follows:
PREPARATION[
]a,c
START
I
I axc
DISCUSSION
I
] a,c
APP-OCS-GEH-322, Rev. D F-1
APIO00 Human Factors Engineering Integrated System Validation Plan (Non-Proprietary)
]a,c
CLOSE
1. Thank participants for their support.
(Last Page of Appendix F)
APP-OCS-GEH-322, Rev. D F-2