Embed Size (px)
Citation preview
This article was downloaded by: [University of North Dakota]On: 03 September 2014, At: 01:38Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number:1072954 Registered office: Mortimer House, 37-41 Mortimer Street,London W1T 3JH, UK
ErgonomicsPublication details, including instructions forauthors and subscription information:http://www.tandfonline.com/loi/terg20
Team performance inprocess control: influencesof interface design andstaffing levelsAngelia Sebok aa OECD Halden Reactor Project, Human FactorsSection, PO Box 173, N-1751 Halden, NorwayPublished online: 10 Nov 2010.
To cite this article: Angelia Sebok (2000) Team performance in process control:influences of interface design and staffing levels, Ergonomics, 43:8, 1210-1236,DOI: 10.1080/00140130050084950
To link to this article: http://dx.doi.org/10.1080/00140130050084950
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of allthe information (the “Content”) contained in the publications on ourplatform. However, Taylor & Francis, our agents, and our licensorsmake no representations or warranties whatsoever as to the accuracy,completeness, or suitability for any purpose of the Content. Anyopinions and views expressed in this publication are the opinions andviews of the authors, and are not the views of or endorsed by Taylor& Francis. The accuracy of the Content should not be relied upon andshould be independently verified with primary sources of information.Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilitieswhatsoever or howsoever caused arising directly or indirectly inconnection with, in relation to or arising out of the use of the Content.
This article may be used for research, teaching, and private studypurposes. Any substantial or systematic reproduction, redistribution,
reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of accessand use can be found at http://www.tandfonline.com/page/terms-and-conditions
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
Team performance in process control: in¯ uences of interface
design and staYng levels
ANGELIA SEBOK*
OECD Halden Reactor Project, Human Factors Section, PO Box 173, N-1751Halden, Norway
Keywords: Human machine interface; StaYng levels; Team performance; Work-load; Situation awareness.
A study performed at the OECD Halden Reactor Project compared the eVects ofinterface design and staYng levels on various aspects of team performance.Teams of nuclear power plant operators participated in challenging simulatorscenarios, working in either a simulated conventional plant, with a hard-controlinterface, or in a simulated advanced plant, with a computerized interface. Two-team staYng levels, normal and minimum, were evaluated in each plantcondition. All teams participated in the same ® ve study conditions, lasting 1 ± 3h each. Several measures assessed team performance: situation awareness,workload, rated team interactions, rated overall performance and objectiveperformance. The ® ndings revealed that combinations of interface design andstaYng levels supported diVerent aspects of performance. Larger crewsconsistently performed better than smaller crews in the conventional plant. Inthe advanced plant, both crew types performed equally well; however, smallercrews had better situation awareness than larger crews. In general, performancewas better for crews using the advanced plant interface, but workload was higher.Workload also was consistently higher in the smaller crews than in the largercrews, regardless of interface type. Links between the performance measures werealso noted.
1. IntroductionHuman machine interaction (HMI) research generally focuses on the individual user
and supporting individual operator performance through appropriate interface
design. However, in reality, teams of operators perform work; crews of pilots ¯ y
planes, groups of medical personnel treat patients, teams of operators run nuclearpower plants. Thus, the technology with which these teams interact must be designed
to support them, not simply to support an individual user.
Much existing HMI research appears to assume that if a design supports
individual performance, it will also support team performance. However, teams have
diVerent requirements than individuals. Teams rely heavily on communication, acommon overview of the situation, and delegation of tasks so that team members
share the workload.
Team performance can be measured in a variety of ways. Previous research has
indicated the usefulness of a variety of measures in assessing performance (Roth et
*e-mail: [email protected]
ERGONOMICS, 2000, VOL. 43, NO. 8, 1210± 1236
Ergonomics ISSN 0014-0139 print/ISSN 1366-584 7 online Ó 2000 Taylor & Francis Ltdhttp://www.tandf.co.uk/journals
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
al. 1994, Sebok et al. 1995). Using a variety of measures allows researchers to capture
a more complete understanding of team performance. Thus, several measures of
team performance were used in this study: situation awareness, workload, team
interactions, rated crew performance and objective performance.
Team performance is related to awareness of the system state; that is, operatorsneed to understand what is happening in the system to take appropriate actions
(Endsley 1995a). Performance is also related to workload. Operators should be
appropriately challenged to perform optimally. If workload is very low, they may be
bored and vigilance will be low, thus resulting in poor performance. If workload is too
high, they may be overburdened and perform ineVectively (Huey and Wickens 1993).Team performance may also be evaluated in terms of team interactions. Teams in
which operators communicate, coordinate activities, and cooperate are more eVective
than teams in which these behaviours are not shown (Coury and Terranova 1991,
Montgomery et al. 1991, Morgan et al. 1986). Furthermore, team performance can be
measured by objectively analysing task completion: if the team accomplishes all the
necessary tasks in a timely manner, they have performed eVectively.In the process control industry, new plant control room interface designs are
being developed to replace existing, conventional technology. These advanced plants
oVer capabilities that are expected to improve crew performance. The interface is
designed to oVer integrated information presented on computerized displays, as
opposed to the scattered information presently in control rooms (ABB 1989, AEC1989, GE Nuclear Energy 1992, Westinghouse Electric 1992). A feature of advanced
control rooms is a large-screen overview display, designed to support team
performance by providing a common reference point for discussions (Roth et al.
1993, Decurnex et al. 1996, Stubler and O’Hara 1996, Roth et al. 1998). Advanced
control room designs, by making plant information accessible from operatorworkstations, allow operators to sit nearer to one another than they do in
conventional plants. This seating arrangement is also expected to improve team
performance by improving verbal and visual communication among team members
(Roth et al. 1993, Fù rdestr ù mmen and Kvalem 1996).
These interface changes, at least in the nuclear industry, are accompanied by
system design changes, e.g. automatic systems and passive plant protection systems(ABB 1989, AEC 1989, GE Nuclear Energy 1992, Westinghouse Electric 1992). This
advanced interface and system technology is expected to so improve crew
performance that fewer operators are expected to be needed in the control room
(ABB 1989). However, these claims must be evaluated.
A study conducted at the OECD Halden Reactor Project addressed the issues ofteam performance in diVerent plant interface and staYng conditions. Team
performance in simulator scenarios was assessed using a diverse set of measures to
capture various aspects of performance. Crews of licensed nuclear power plant
operators participated in realistically simulated conditions to identify the eVects of
interface type and staYng con® guration on diVerent aspects of team performance.
1.1. Interface type
The proposed interface design changes in advanced plant control rooms are expected
to improve team performance. However, they could, in fact, have a negative eVect on
team performance, and thus need to be evaluated. The following sections describe
how diVerent aspects of crew performance may potentially be aVected by changes ininterface design from conventional to advanced.
1211Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
1.1.1. Situation awareness: Interfaces that provide integrated information and a
common overview display are expected to support team situation awareness.
Situation awareness is de® ned as t̀he perception of elements in the environment
within a volume of time and space, the comprehension of the meaning, and the
projection of their status in the near future’ (Endsley 1988b). Designers anticipate thatthe team situation awareness will be supported by displays that integrate information.
Conventional plants require operators to look at individual displays containing low-
level information such as valve position, and then to integrate the individual pieces of
information. In contrast, the advanced plants typically provide an overview and
displays that integrate the information for the operator. Thus, the operators’situation awareness should be better supported by the advanced plant interface.
Another factor in advanced plant design expected to support operator situation
awareness is the presence of a common overview display. By providing a common
reference point which will form a basis for discussions, the overview display should
support operators’ situation awareness (Stubler and O’Hara 1996, Roth et al. 1998).
However, advanced plant designs that present high-level information couldactually have a negative eVect on team situation awareness. Crews may rely on the
individual pieces of information and the mental act of integrating the information
may be what builds situation awareness in a team. Perhaps having the information
already integrated may make the operators complacent and lower overall awareness
of the plant state. The act of browsing or scanning, a task that conventionaltechnology supports better than advanced appears to (Woods 1984), may support
situation awareness. In using advanced plant interfaces, operators lacking
information obtained by visual scanning may have impaired situation awareness.
If the system presents integrated information in a `predigested’ form, the meaning of
the data may be obscured (O’Hara and Hall 1992). Also, changing the role of theoperator from an active information gatherer to a passive monitor may adversely
aVect situation awareness (Idaszak 1989).
The eVects of increased automation on team and plant performance are
uncertain. Perhaps automation improves performance by supporting the operator
when necessary, but it could cause problems by taking the operator out of the loop.
Often, the tasks automated in system design are those that engineers can easilyautomate (Moray, personal communication, 1997). Because they are not analysed
from the perspective of the user, they are not necessarily tasks that the operator
needs or wants taken from his or her control. This improper allocation of tasks to be
automated can result in the problem of clumsy automation that could have a
negative impact on team and plant performance.
1.1.2. Workload: Perceived workload is important in nuclear power plant control
rooms because of its eVect on human error and performance. In general, a moderate
workload is considered optimal. If workload is much below this optimal level,
operators become under-stimulated, may become inattentive, lose situationawareness and suVer degraded performance when the operating condition demands
increased eVort. Similarly, deviating from the optimal in the other direction, by
overloading operators, has a negative impact on performance. Operators with too
high a workload begin postponing less-critical tasks to complete the most critical
tasks. Between these two extremes is the optimal workload that suYciently
challenges and stimulates operators without overtaxing their capabilities (Hueyand Wickens 1993).
1212 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
Advanced plants are being designed with the intent of lowering workload by
improving the control room interface design and increasing automation. As
automatic systems take over tasks in the plant, and information is accessible by
computer displays, the work done by individual operators is expected to decrease.
However, workload could be increased for operators in the advanced plant, asthey attempt to cope with computerized systems and work in a new, unfamiliar type
of plant. The cognitive eVort in managing the interface, navigating through
computer workspaces, and having to retrieve information rather than having it
consistently present may all contribute to higher workload for teams working with
advanced plant interfaces (O’Hara and Hall 1992, Schryver 1994).
1.1.3. Team interactions: Team interactions, i.e. how well the team functions as a
unit, are also expected to be improved in advanced plants. The cockpit design of new
control rooms is expected to support team interaction by positioning the operators
nearer one another and allowing the operators to see easily what their fellow team
members are doing (Roth et al. 1993, Fù rdestrù mmen and Kvalem 1996).Furthermore, team interactions should be improved in advanced plants by the
shared overview display. The overview is expected to facilitate communication,
coordination, and cooperative problem solving (Roth et al. 1998). By enhancing
these aspects of team interaction, overview displays are expected to contribute to
improved team performance in advanced plants (Roth et al. 1993, Decurnex et al.1996, Roth et al. 1998).
However, issues such as distribution of responsibility and crew dynamics must
also be considered. Team interactions may be too complex to be supported simply by
positioning operators near one another and providing a common overview display.
1.1.4. Overall performance: By supporting situation awareness, lowering workload,
and enhancing team interactions, advanced plant design is expected to improve
overall team performance. In addition, the automatic systems and passive plant
protection systems are expected to provide a back-up support for operators, thus
further enhancing overall plant performance.
1.2. StaYng levels
StaYng levels, the number of team members available to work in a control room,
may have an impact on performance. The eVects of changes in staYng levels are
perhaps even less predictable than changes in interface design. As crews are reduced
from four to two members, signi® cant eVects may be expected on performance.These will be considered individually.
1.2.1. Situation awareness: The eVect of staYng levels on team situation
awareness is unclear, due in part to a lack of consensus on the de® nition of
team situation awareness. A couple of opposite possibilities exist: crew situationawareness could be synergistic or t̀he whole is greater than the sum of the parts’
(Regal et al. 1988), or it could be de® ned by the `weakest link’ (Endsley 1995a). If
crew situation awareness is in fact synergistic, larger crews would support situation
awareness by having more people available to assimilate information. In a large
team of operators, if one has good situation awareness, overall crew situation
awareness can be improved by communicating with the other team members (Hueyand Wickens 1993).
1213Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
However, another possibility is that crew situation awareness is actually a `weakest
link’ concept; if even one member of the crew lacks awareness of the process state, the
overall crew situation awareness is degraded to that level (Endsley 1995a). Thus,
having additional people in the control room crew would increase the possibilities for a
weakest link. That is, the more people in the control room, the higher the probability ofhaving someone who is confused about the state of the plant. This operator with low
situation awareness would then become the weakest link.
1.2.2. Workload: The eVect of crew staYng on workload must also be evaluated.
In the design of advanced plants, vendors expect operator workload to be decreaseddue to the improved interface and automatic systems. This workload decrease is
expected to be signi® cant enough that vendors anticipate that two-person crews
could safely control the plant (ABB 1989). The eVect of this anticipated staYng
reduction on operator workload is not explicitly addressed in vendor design
proposals, but the implicit assumption appears to be that the amount of operator
workload in advanced plants is comparable with workload in conventional plants.This issue must also be addressed.
One possibility, following the logic of the vendor design proposals, is that,
because of the improved system design, operator workload is signi® cantly reduced in
advanced control rooms. In order to keep operators suYciently challenged and
attentive, staYng levels may need to be reduced to allow operators additionalresponsibilities. If this is true, an extra person or people in the advanced control
room could, by taking over tasks, lower individual operator workload to a
suboptimal level, so performance is actually degraded. Another, more likely
possibility is that the extra person (or people) has no clear role and does not
eVectively contribute to lowering team workload.However,workloadmaynotbe reducedintheadvancedplants.Theinterfaceand the
newroleof theoperatingcrewwithrespect to theautomaticsystemsallmaycontribute to
higher workload. Reducing the staYng level of the crew may result in even higher
individual workload, with too few people available to perform the necessary tasks.
1.2.3. Team interaction: The eVect of staYng levels on team interaction is alsouncertain. Reduced staYng levels may have a negative impact on team interactions
by making interpersonal dynamics too important. In a team of operators, one
dominant person may take over, but more people are available to question him/her.
In a crew of two operators, or in small crews, the dominant person may take over
and the less dominant may simply comply (Foushee 1984, Foushee and Helmreich1988). These types of interactions have been a serious problem resulting in aviation
accidents, when the dominant person has mistaken beliefs about the state of the
system and insists that other crew members comply with his/her diagnosis and
remediation strategy (Foushee 1984).
However, another possibility is that, in larger crews, a feeling of distributedresponsibility prevents individual team members from disagreeing on a faulty
diagnosis. In larger crews, individual team members who disagree with the group
diagnosis may be either afraid to disagree or may assume that another person will
take the responsibility of disagreeing (Foushee 1984).
1.2.4. Overall performance: The eVects of staYng levels on overall teamperformance are uncertain. Team performance may be enhanced with more people,
1214 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
as more people are available to contribute to diagnosis formation and to perform
tasks needed to ensure plant safety. However, more people in larger crews may create
more possibilities for error, more confusion, a sense of distributed responsibility, and
a stronger dependence on crew coordination.
2. Methods
2.1. Design
To investigate the eVects of interface design and staYng levels on team performance,
an experiment was conducted using a between-subjects design with eight crews of
operators participating in ® ve simulated scenarios. Half of the crews participated inthe advanced plant condition, half of them participated in the conventional plant
condition.
Two independent variables were staYng levels and interface type. Crews of two
diVerent staYng levels participated in each of the two interface conditions. These
con® gurations included a normal staYng complement and minimal complement. In
both experimental plant conditions, the normal staYng level was four operators. Theminimal staYng level in a conventional plant was three operators; in the advanced
plant, two operators were the minimum staYng level. Table 1 shows the staYng
levels in the diVerent plant conditions.
To assess team performance in the scenarios, ® ve measures were taken to
evaluate diVerent aspects of performance: situation awareness, workload, teaminteractions, rated crew performance and objective performance.
2.1.1. Situation awareness: Situation awareness was measured using the inter-
ruptive questionnaire technique developed by Endsley (1988b) for aviation. It was
modi® ed especially for the process-control industry (Hogg et al. 1995). Previousresearch has shown the interruptive questionnaire technique to be valid and
relatively non-intrusive (Endsley 1995b).
The situation awareness inventory used in this study is the Situation Awareness
Control Room Inventory (SACRI) developed by Hogg et al. (1995). The questions
are speci® c to nuclear power plant control and ask about trends of diVerent process
parameters. Questions ask about the past, present and future state of both primaryand secondary side parameters, with equal representation across time periods and
plant systems.
The de® nition of crew situation awareness in this study was the average situation
awareness of the individual members. That is, the individual crew members’ situation
awareness was measured and the average of their results was taken to be the crew’ssituation awareness. Reasons for choosing this technique were the disagreement
regarding the de® nition of crew situation awareness (Endsley 1995a, Regal et al.
1988) and the relative simplicity of the individual measure.
Table 1. StaYng levels in the advanced and conventional plants.
StaYng levels Interface typeAdvanced plant Conventional plant
NormalMinimal
four-persontwo-person
four-personthree-person
1215Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
2.1.2. Workload: NASA TLX, a commonly used and validated metric for assessing
individual workload (Hill et al. 1992, Weirwille and Eggemeier 1993), was the
workload assessment inventory in this study. This questionnaire has rating scales to
assess six components of workload: mental demand, physical demand, temporal
demand, performance, eVort and frustration. Participants rated workload for eachcomponent, from 0 to 100, with 0 being `no workload’ and 100 being `highest
possible workload’. The individual operators’ ratings of perceived workload were
averaged to give the team workload.
2.1.3. Rated team interactions: Team interactions were rated by means of aBehaviourally Anchored Rating Scale (BARS) technique. This particular measure
was developed based on previous research (Montgomery et al. 1991) and in
cooperation with process experts to identify anchors, or critical behaviours
indicating good team interactions. The ratings were task focus/decision-making,
coordination as a crew, communication, openness, and team spirit. Each of these
was identi® ed with several example behaviours (positive and negative) for raters toobserve and use as criteria in their rating. The overall rating was a number between 1
and 7, with 7 being the best team interactions.
2.1.4. Rated overall performance: Three process and training experts rated team
performance in the scenarios. The rating criteria were: solution path, control ofplant, communication, and con® dence (Hanson et al. 1987). Solution path refers to
the crew’ s use of time in recognizing the event and selecting the correct mitigation
procedure(s). Control of plant refers to the crew’s understanding of procedures in
their analysis of the transient and the extent to which they challenged safety
equipment. Communication refers to the extent to which the information exchangeamong the crewmembers facilitated transient mitigation. Con® dence refers to the
ease with which the crew completed transient mitigation without hesitation, and self-
statements about the sureness of their own actions and decisions. Each dimension
was rated on a scale of 1 ± 10, with 1 being the worst performance, and 10 being
optimal performance.
2.1.5. Objective team performance: Team performance in each scenario was also
evaluated in terms of objective measures: announcements and noti® cations,
important task performance, and stabilization and cooldown.
In all scenarios, crews were required to make plant-wide announcements and oV-
site noti® cations. The necessary communications were identi® ed for each scenario(e.g. notifying emergency response teams, local and national authorities; making
plant-wide ® re announcements) and the crews were evaluated according to whether
they made these announcements and noti ® cations.
Identifying the critical tasks necessary for handling the main fault in the scenario
assessed important task performance. The necessary mitigation actions diVered ineach of the scenarios. For each scenario, these actions were identi® ed, and crew
performance compared with these criteria. As an example of an important task, in a
steam generator tube rupture (SGTR) scenario, the critical tasks included diagnosing
the SGTR and isolating the faulty steam generator.
Cool-down and stabilization activities provided a third indicator of objective
crew performance. All scenarios were of suYcient duration to allow the crews time tobegin stabilizing activities. Cool down rates, boiling margins, inventory levels in
1216 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
safety and auxiliary systems, and pressure diVerentials had to be maintained within
target ranges to prevent unintended challenges to equipment and systems. The ability
of the crew to keep the plant parameters within target ranges was evaluated.
2.2. Participants2.2.1. Teams of operators: Eight crews of licensed nuclear power plant operators
from the Loviisa nuclear power plant in Finland participated in the study. These
crews varied in number of personnel, from two to four per crew. A total of 26
operators participated in the study. Data on age were not collected, and no attempt
was made to balance crews in terms of years of experience or education. Crews were,to the extent possible (also due to availability constraints), actual operating crews at
the plant. At the Loviisa plant, the practice was to have three-man crews, so the four-
man and two-man conditions represented modi® ed crews.
All crews consisted of, at a minimum, a reactor operator and a turbine operator.
In the three- and four-man crews, a shift supervisor was also included. In the four-
man crews, the fourth person was a licensed operator with the role of a control roomassistant. In the two-man crews, either the reactor operator or the turbine operator
served as a dual role operator and shift supervisor.
2.2.2. Raters: Three process experts participated in the experiment as performance
raters. These personnel had between 10 and 30 years of nuclear process experience.They observed crew performance throughout the scenarios and rated various aspects
of performance. Two of the raters were in the control room with the operators
(unobtrusively positioned in the corner of the room) and one was in the instructor
gallery.
2.2.3. Expert commentator and communications: A training specialist from the
Loviisa plant, the rater in the instructor gallery, acted as an expert commentator and
provided role-playing as external personnel. When the control room operators made
oV-site calls or radio calls to a ® eld operator, the expert commentator role-played the
part of the external specialist. The expert commentator recorded the phone calls
made by the crew.
2.2.4. Simulator instructor: Two simulator instructors, one at each of the two plant
simulators, participated in the experimental study. The instructors’ role was to start
and stop the simulator, initiate planned events, and monitor crew actions. The
instructor received training on the scenarios and communicated with theexperimenters and expert commentator regarding the progress of the scenarios.
2.3. Apparatus
2.3.1. Simulators: Two diVerent types of simulators were used in the study: one
simulator represented a conventional plant with a hard-panel interface and the othersimulator provided an advanced interface with computerized displays. Both
simulators were based on the same plant model, the Russian-style Pressurised
Water Reactor nuclear power plant in Loviisa, Finland. Both simulators included an
instructor gallery, where the experimenters could watch the experiment in progress.
The simulators were chosen because they represented the same plant model and
oVered diVerent interfaces, representative of conventional and advanced planttechnology.
1217Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
The conventional plant, simulated at the Loviisa Training Facility at the Loviisa
nuclear power plant, is shown in ® gure 1. This plant interface consists of
conventional controls and displays. Three operator workstations are provided.
Operators are seated several meters apart from each other, and they frequently walk
around the control room to read indicators and observe plant status.The conventional plant was a replica of the actual control room in which the
operators worked. The training simulator for the conventional plant actually has a
hybrid type of plant interface, with advanced technology integrated with
conventional displays and controls (e.g. meters, light indicators, tile alarms). In
this study, the computer support systems at the Loviisa training facility weredisabled, and the steam generators were modelled as having one-third the actual
capacity, to make the plant more representative of a typical conventional-style
plant.
The advanced plant was simulated in the Halden Man ± Machine Interaction
Laboratory (HAMMLAB), in Halden, Norway. The advanced plant condition
(® gure 2) featured a fully computerized interface, with trend diagrams, list alarms,and a common overview display. In the advanced plant interface, operators were
seated within a few meters of one another and were able to view, if not read, each
other’ s displays. Operators did not need to walk around the control room; rather, the
design encouraged them to remain seated.
2.3.2. Data-collection equipment: Several types of equipment collected data in the
study: videotapes with audio recordings, paper questionnaires, and simulator
records. Videotapes were made of the crew’s performance throughout the entire
scenario, with three cameras recording the crew’s actions from diVerent perspectives.
These videotapes were time-stamped to allow comparisons with simulator logs.
Figure 1. Conventional plant interface (Loviisa, Finland).
1218 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
Audio recordings were made on separate tracks of the videotape: one track included
individual crewmember verbalizations, the other track included a simulator
instructor’s comments on and interpretations of crew performance. Questionnaireswere used for collecting situation awareness, workload, team interaction, and rated
team performance data. Two simulator records, a variable log and an event log, were
made for each scenario. The variable log contained values of 100 diVerent plant
parameters, recorded every 15 seconds. The event log recorded simulator instructor
and operator actions, as well as major process events (e.g. automatic safety system
initiation, alarm status). These logs contained time stamps for comparison with thevideotape.
2.4. Experimental procedure
2.4.1. Training: All experimental participants (i.e. crews of operators) received
appropriate training for the simulator condition in which they participatedduring the study. In the case of the conventional plant simulator, the operators’
own plant training simulator, operators received ~ 2 h of training on the
changes made to the instrumentation and control systems and plant performance
characteristics to more closely approximate a conventional plant. In the
advanced plant condition, simulated in HAMMLAB, operators received ~ 2.5days of training on the advanced plant characteristics and the computerized
simulator interface.
Participants also received training on the data collection inventories to be
completed during the study. They were asked to complete the questionnaires as
accurately as they could. They were also requested not to discuss the scenarios with
other operators at the plant, as any foreknowledge of the conditions mightcontaminate results.
Figure 2. Advanced plant interface (Halden, Norway).
1219Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
Raters of team interactions received training on the inventories for team
interaction and rated crew performance. The raters participated in designing the
team interaction inventory, which they ® eld tested during a pilot study session.
Following training, the crew was allowed a 15 ± 30-min break before participating
in the experimental scenarios.
2.4.2. Scenarios: All crews of operators participated in ® ve multiple-fault scenarios.
These varied between 1 and 3 h in duration. The scenarios were presented in random
order. Experimental scenarios were conducted over 2 days for each crew (for a total of
16 days of experimental scenarios), with a maximum of three scenarios per day.All ® ve scenarios required operators to perform mitigation actions to control the
plant and required some degree of coordination with personnel outside the control
room, as simulated by the training specialist. The ® ve scenarios are identi® ed below,
by the critical fault that occurred in the scenario.
· Steam generator tube rupture (SGTR).
· Interfacing systems loss of coolant accident (ISLOCA).
· Loss of feed water (LOFW).
· Loss of oV-site power (LOOP).
· Steam generator over® ll (SGOF).
All scenarios were divided into either four or ® ve distinct periods. During the ® rst
period, teams performed simple tasks or monitored the plant status. During the
second and third periods, disturbances were initiated and alarms sounded in the
control room. During the fourth and ® fth period, new disturbances were generally
not initiated (depending on the team’s performance and handling of the scenario).Periods were ~ 15 ± 30 min each.
2.4.3. Order of events: Following training and the break, operators were seated at
their respective workstations in the simulator. Raters took their seats either in a
corner of the simulated control room or in the instructor booth.
The crew of operators received a short brie® ng on the present state of the plant,appropriate for the particular scenario, and were given a work order for a routine
task. The two raters in the control room were given team interaction rating forms so
they could compare crew performance against the behavioural anchors established
on the inventory.
The simulator instructor and other laboratory technicians initiated the datacollection systems (i.e. videotapes, microphones, simulator and simulator data logs).
An experimenter announced to the crew that the simulator was running.
The simulator was run for the entire period. Crews took actions as they would
during normal operations and the simulator responded accordingly, within the
constraints of the scenario. At the end of each period, the simulator was frozen. Thefreeze was announced (i.e. t̀he simulator is frozen’), and data collection inventories
were distributed to the crewmembers.
During the pause, operators turned away from the monitors or control panels,
and completed the situation awareness and workload questionnaires. While
operators completed their inventories, the two raters evaluated the team’s
interactions during just-completed scenario period. The pauses were not timed;operators and raters were allowed to take as long as necessary to complete the
1220 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
questionnaires. However, they had been informed prior to the study to complete the
questionnaires as quickly as possible. Pauses typically lasted ~ 5 min, to allow all
participants suYcient time to complete their respective data-collection inventories.
When the data collection inventories were collected and veri® ed complete, the
simulator was restarted. This process was repeated until the entire scenario wascompleted (i.e. four or ® ve periods). At the end of the scenario, the operators and
raters completed a ® nal set of data collection inventories. Following this task, the
experimental participants took a 15 ± 30-min break before resuming another
scenario.
On completion of a scenario, three raters (the two-team interaction raters and atraining instructor) evaluated the team’s overall performance on the scenario. After
the experimental sessions were complete, an experimenter and a process expert
evaluated the team’s objective performance in each scenario. Table 2 summarizes the
data collection conducted throughout the experimental scenarios. Table 2 identi® es
the diVerent measures collected, the technique, the type of data, the person (or
people) providing the data, and when the data were collected.
2.5. Analyses
Data were analysed to determine the eVects of plant type and crew con® guration on
the various performance measures. Analyses were performed using Statistica Ò
(StatSoft, Inc.) software.Measures were evaluated individually. Measures taken throughout a scenario
(i.e. situation awareness, workload, team interaction) were averaged across
crewmembers (or raters, in the case of the team interaction data) and teams, and
plotted against time to reveal general trends throughout the scenario. Then, the data
for each measure were evaluated in more detail to compare crew performance in thevarious interface types and staYng levels.
For situation awareness, workload and team interactions, ANOVA were
conducted to identify signi® cant (p < 0.05) diVerences in performance among crews
in the various plant interface types and staYng levels (Winer 1971).
Rated crew performance and objective performance were collected once per
scenario rather than at intervals throughout the scenario. These data were evaluated
Table 2. Summary of data collection techniques.
Measure Technique Type Person When
Situation awareness
Workload
Team interaction
Rated teamperformance
Objective performance
SACRI
NASA TLX
BARS,developed forthe study
developed byHanson et al.(1987)
developed forthe study
questionnaire
rating scales
rating scales
rating scales
performancecriteria
individualoperators
individualoperators
two trainedraters
three trainedraters
one processexpert, oneexperimenter
each period
each period
each period
each scenario
each scenario
1221Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
by calculating means of performance ratings for the diVerent interface types and
staYng levels. Signi® cance tests were conducted to identify signi® cant (p < 0.05)
performance diVerences among crews in the various interface type and staYng level
conditions.
3. Results
3.1. Summary of performance measures
Table 3 presents the diVerences identi® ed in crew performance based on interface
type, staYng level, and their interaction. All ® ve of the performance measures are
presented to facilitate comparisons. The p-value for each signi® cant eVect isidenti® ed, together with the means of crew performance. These means were
calculated by collapsing across all crews in the respective conditions, as well as all
periods (i.e. for situation awareness, workload, and team interaction). All cells
identi® ed with two hyphens (--) indicate that no signi® cant eVect was identi® ed for
that particular condition.
3.2. Situation awareness
3.2.1. EVects across scenario periods: Figure 3 presents the average situation
awareness of all crews across scenario periods. At the beginning of the scenarios,
crews had a relatively high level of situation awareness that dropped when scenario
disturbances were initiated. Later in the scenarios, crews began recovering situationawareness.
Table 3. Crew performance diVerences on all measures identi® ed for interface type, staYnglevel and their interaction.
Measure Interface type StaYng levelInterface type ´staYng level
Situation awareness
- - - -
p= 0.0217
Adv. Nor: 0.649Adv. Min: 0.738Con, Nor: 0.771Con, Min: 0.602
Workload p= 0.0006
Adv: 48.14Con: 38.19
p= 0.0231
Nor: 40.30Min: 46.03
- -
Team interaction p= 0.0010
Adv: 5.46Con: 4.49
- - - -
Rated crewperformance
p< 0.0001
Adv: 7.74Con: 6.44
- -
p= 0.0004
Adv, Nor: 7.47Adv, Min: 8.02Con, Nor: 7.30Con, Min: 5.58
Objectiveperformance
p= 0.0269
Adv: 3.62Con: 2.42
- - - -
1222 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
3.2.2. Interaction of interface type and staYng level: Crew situation awareness
varied depending on the interaction of interface type and the staYng level. In the
conventional plant interface condition, the normal sized crews had better situationawareness. However, in the advanced plant interface condition, the smaller crews
had better situation awareness. These diVerences were signi® cant during the third
and fourth periods of the scenario (F(1,75)6.8261, p = 0.0109; F(1,75)8.0119,
p = 0.0060 respectively).
To examine these diVerences more closely, ® gure 4 shows the average
situation awareness across scenario periods for the two interface types andstaYng levels. The normal-sized crews in the conventional plant and the
minimum-sized crews in the advanced plant exhibited stable, consistent situation
awareness. The minimum-sized crews in the conventional condition showed
decreasing situation awareness, but recovering in the ® fth period. In the
advanced plant, the normal-sized crews also exhibited decreasing, but recovered,situation awareness.
3.3. Workload
3.3.1. EVects across scenario periods: Figure 5 shows the change in perceived
workload across scenario periods, averaged across all crews. Workload began at arelatively low level, increased dramatically with the initiation of disturbances, and
remained at this higher level throughout the scenario.
3.2.2. Interface type: Subjective workload varied depending on the interface type,
being rated higher in the advanced plant condition (® gure 6). DiVerences in
subjective workload for the two interface types were revealed during periods 2 ± 5(F(1,71)18.7399, p = 0.00005; F(1,71)10.1581, p = 0.0021; F(1,71) 20.3827, p = 0.00003;
Figure 3. Plot of average situation awareness by scenario period.
1223Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
Figure 4. Plot of situation awareness by interface type and staYng level, across scenarioperiods.
Figure 5. Plot of average subjective workload across scenario periods.
1224 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
F(1,71)26.6055, p < 0.00001). The increase in workload following the onset of
disturbances was greater in the advanced plant than in the conventional plant.
3.3.3. StaYng level: DiVerences in subjective workload were also found between
the staYng levels (® gure 7). During periods 1, 2 and 5, the minimum-staV crewsreported signi® cantly more workload than normal-staV crews (F(1,71) 4.9714,
p = 0.0289; F(1,71) 9.3891, p = 0.0031; F(1,71) 4.0842, p = 0.04706).
3.4. Rated team interactions
Inter-rater reliability was evaluated using the Pearson product moment correlationcoeYcient (r = 55). The diVerence between the two raters was due to variation in
rating the subscales in the individual crew interaction ratings.
3.4.1. EVects across scenario periods: Figure 8 shows the change in average team
interaction across the scenario periods. Team interaction began at a moderate level,
increased slightly when disturbances were initiated, then dropped throughout theremainder of the scenario.
Figure 6. Plot of subjective workload by interface type, across scenario periods.
1225Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
Figure 7. Plot of subjective workload by staYng level, across scenario periods.
Figure 8. Plot of average team interaction across scenario periods.
1226 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
3.4.2. Interface type: Team interactions were rated higher in the advanced plant than
in the conventional plant. No diVerence in team interaction was found between the
staYng levels, and no interaction eVect of interface type and staYng level was revealed.
Figure 9 presents the changes in team interaction ratings for the two interface
types, and shows that team interaction was rated as better in the advanced plant thanin the conventional plant.
These diVerences were signi® cant in periods 1 and 3 ± 5 (F(1,42) 8.4416,
p = 0.0058; F(1,42) 9.3249, p = 0.0039; F(1,42) 11.5145, p = 0.0015; F(1,42) 23.1192,
p < 0.0001).
3.5. Rated overall performance
Inter-rater reliability was evaluated. The Pearson product moment correlation
coeYcient of 0.65 was obtained.
3.5.1. Interface type: Crews in the advanced plant were rated as having better
overall performance than crews in the conventional plant, as determined bycomparing means in a two-tailed signi® cance test.
3.5.2. Interaction of interface type and staYng level: An interaction eVect was also
observed, as summarized in table 5 and presented in ® gure 10. Performance in the
conventional plant, minimum-sized crews was rated lower than the other threeinterface and staYng level conditions.
Figure 9. Plot of team interaction by interface type, across scenario periods.
1227Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
3.6. Objective team performance
3.6.1. Interface type: Crews in the advanced plant performed important tasks and
cooldown and stabilization tasks signi® cantly better (p < 0.0001 for both task types)than crews in the conventional plant (® gure 11). No diVerences were observed
Table 4. Rated overall performance in the interface type conditions.
Conventional Advanced p
Average rated performance 6.438 7.742 0.0001
Table 5. Signi® cance levels in interface type ´ staYng level conditions.
Condition Conv. normal Conv. minimum Adv. normal Adv. minimum
Conv. normalConv. minimumAdv. normalAdv. minimum
x0.00040.64150.0596
0.0004x
0.0001< 0.0001
0.64150.0001
x0.1200
0.0596< 0.0001
0.1200x
Figure 10. Average rated crew performance in the interface type and staYng level conditions.
1228 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
between interface types regarding announcements and noti® cations. No diVerenceswere revealed among the two staYng levels.
4. Discussion
4.1. Changes across scenario periods
Measures taken throughout a scenario (i.e. situation awareness, workload, teaminteractions) varied across scenarios in a predictable manner. Situation awareness
began at a high level, dropped when the plant disturbances began, and increased
again towards the end of the scenario. Workload began at a low level, increased
when disturbances were initiated, and remained at a high level throughout the
scenarios. Team interactions began at a relatively moderate level, increased when thedisturbances began, and dropped later in the scenario.
These ® ndings are easily understood, considering the progression of events in a
scenario. The ® rst period of the scenarios contained a simple task; disturbances had
not yet begun. Crews were easily able to perform routine tasks. They were aware of
the state of the plant and they had low workload.
During the second and third periods, various disturbances were initiated. Whendisturbances began and the crew started receiving alarms in the control room, the
Figure 11. Average objective crew performance measures by interface type.
1229Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
situation changed. Situation awareness dropped: crews were uncertain as to the state
of the plant. Workload increased as crews began diagnosing and handling the
problems. Team interactions improved as the crew began communicating and
sharing information to diagnose problems. During the second and third periods,
crews worked together to identify, diagnose, and solve the problems. They allocatedtasks and began them.
The crews’ actions determined the amount and type of disturbances in the fourth
and ® fth scenario periods. New disturbances generally did not occur during that part
of the scenario. In these ® nal periods, crews no longer worked together so eVectively:
they were busy performing their individual tasks. Their workload remained high asthey went about their tasks. As crews diagnosed and worked to mitigate the plant
disturbances, situation awareness increased.
4.2. Interface type
For many of the measures (e.g. team interaction, rated crew performance, and
objective crew performance), team performance was better in the advanced plantthan in the conventional plant. The improvements in team interaction and rated
crew performance in the advanced plant interface condition were noted for both
the minimum and the normal sized crews, but were more pronounced in the
minimum crews. The objective crew performance was better in the advanced plant
condition for critical task performance and for stabilization and cooldown.Workload was higher in the advanced plant than in the conventional plant, for
both types of crews.
The ® nding of better performance in the advanced plant is particularly
interesting, because it was a diVerent simulator than the crews’ normal operating
simulator. Crews in the advanced plant condition needed to learn to use the interfaceas well as receive training on the changes to the process model. Despite the
additional training requirements, crews in the advanced plant condition performed
better than crews in the conventional plant condition.
The increased workload in the advanced plant may be in part due to the
additional stress of working in an unfamiliar simulator. If the only in¯ uencing factor
were the familiarity with the interface type, the workload should be higherthroughout the entire scenario. However, it was equivalent for the baseline period
(i.e. period 1), increasing dramatically only after the disturbances were initiated.
The diVerences in workload between the two interface conditions indicate that
operators felt better able to handle disturbances in the conventional condition (i.e.
their own simulator) than in the advanced plant condition. Given the trainingoperators had using the two interface types, years of training using the
conventional type of interface versus 2 days of training on the advanced interface,
the diVerences in perceived workload with the two interface types are surprisingly
minor.
Team interaction was rated as signi® cantly better in the advanced plant than inthe conventional plant. The design of the advanced plant simulator, placing the
operators near one another and providing a common overview display, appears to
support team interaction and performance. However, these design features can not
de® nitively be stated to be the cause of improved performance. In the advanced plant
condition, operators communicated more frequently and had more positive
interactions, and they have a common reference point for their discussions in theoverview. These ® ndings agree well with other study results (Roth et al. 1993, Stubler
1230 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
and O’Hara 1996, Decurnex et al. 1996, Roth et al. 1998). Another possible reason
for the improved team interaction may be the perceived need to cooperate and
communicate in the new simulator, as opposed to the operators’ own familiar
simulator.
Further, these results indicate that the advanced plant interface supported ratedand objective performance. Crews in the advanced plant condition were rated as
having better performance and, objectively, they were better able to perform their
critical tasks as well as to perform stabilization and cooldown tasks in the advanced
plant condition. No signi® cant changes were noted regarding the announcements
and noti ® cations for the two plant types. However, announcements and noti® cationsdo not directly depend on the interface: they depend more on the crew identifying
and ® nding time to make the necessary phone calls and radio announcements.
Interface-dependent tasks that require controlling the state of the plant (i.e.
important task performance and stabilization and cooldown) were better handled by
using the advanced plant interface than the conventional plant interface. Another
possibility, not addressed in this study, is that the improvements in crewperformance in the advanced plant condition were due to the system features (e.g.
automatic systems) rather than the interface.
4.3. StaYng levels
The staYng levels of the various crews aVected various aspects of team performance.Workload was higher in the smaller crews, regardless of interface type. Intuitively,
this ® nding makes sense: the more people available to perform the work, the lower
the workload.
4.4. Interaction of interface type and staYng levelIn the conventional plant, the normal-sized crews (i.e. the four-person crew)
performed better than the minimal-sized crews (i.e. the three-person crew). Situation
awareness was higher; workload was lower; and rated crew performance was better
for the normal sized crews than for the minimal sized crews.
This ® nding was surprising, because the normal operating practice at this
particular plant is to have three operators on duty, so the four-person crew actuallyrepresented an overstaVed condition. However, this additional person improved
performance in the simulated scenarios. The extra person was available to help
gather information and build situation awareness, to share tasks and lower
workload, to contribute to discussions and improve team interactions and support
team performance.In the advanced plant conditions, little diVerence was found between the two
crew sizes. Crews performed equally well in terms of team interaction, rated crew
performance, and objective performance. In the advanced plant, the extra
crewmembers were of limited bene® t. They did not signi® cantly reduce workload,
nor did they improve team performance, rated performance, or objectiveperformance. The extra crewmembers did not contribute to situation awareness:
smaller crews had better situation awareness. However, the extra crewmembers did
contribute to reducing workload. Even so, both crew staYng levels in the advanced
plant had higher workload than crews in the conventional plant condition.
Advanced plants appear designed to allow a smaller crew (i.e. two operators) to
handle the plant in terms of performance issues (i.e. situation awareness, teaminteraction, rated crew performance) as well as, or better, than larger crews.
1231Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
4.5. Links among performance measures
Previous research has found that situation awareness is necessary, but not suYcient,
for eVective performance (Endsley 1995a). This study supports that ® nding, to an
extent. In the conventional plant, situation awareness was higher for the normally
staVed crews. These crews were rated as having higher performance, compared tominimum crews in the conventional plant. Similarly, in the advanced plant, situation
awareness was higher in the minimally staVed crews and performance was rated
higher. However, no signi® cant diVerences were found between the conventional and
advanced plants in terms of overall situation awareness and yet signi® cant
performance diVerences were noted between the two plant interface types. These® ndings indicate, as Endsley (1995a) predicts, that situation awareness is related to
performance, but is not the only factor in¯ uencing performance.
Workload and performance are typically believed to be related, roughly, by an
inverse U-shaped function, with an optimal level of workload results in optimal
performance (Huey and Wickens 1993). However, the point at which workload
degrades performance is unknown and sudden: when the workload becomes toohigh, performance may be maintained temporarily. Then, suddenly, performance
drops signi® cantly and dramatically (BengstroÈ m 1993). In this study, crews in the
advanced condition performed better yet experienced higher workload, thus
suggesting that workload was high enough to be challenging, but not so high as
to degrade performance.Previous research has indicated that team interactions, or eVective performance
as a team, leads to better objective performance (Morgan et al. 1986, Glickman et al.
1987, Montgomery et al. 1991, Roth et al. 1998). The same ® nding appeared in this
study. The teams rated as having the best team interactions, the crews working in the
advanced plant condition, had higher rated performance and higher objectiveperformance. The crew with the lowest rated team interactions, the minimum crew
con® guration using the conventional interfaces, had the lowest rated and objective
performance. These ® ndings also indicate a link between team interactions and task
performance.
4.6. Comments on methodology and generalizabilityIn experimental studies, a balance must be achieved between experimental control
and generalizability of results (Cook and Campbell 1979). This study provided
experimental control through scenario design, selection of crew staYng levels and
simulator interface conditions, and equivalent treatment during the experiment.
Generalizability requirements were addressed by providing a high degree ofrealism. Licensed nuclear power plant operators participated in the study,
completing ® ve scenarios in a large-scope, high-® delity simulator. The scenarios
were made realistic by their long duration, the crews’ freedom to take actions as
they would in the plant, and the simulation of outside-the-control-room
communications.However, in designing a realistic experiment, a degree of experimental control
may be lost. Experimental conditions, scenarios, were intentionally designed so the
major events (e.g. faults, alarm indications) would not depend on operator actions,
but could occur independently. Because crews were able to take actions as in the
plant, all crews participated in somewhat diVerent scenarios. The general course of
events in the scenarios were identical, so the between-crew scenario diVerences wererelatively minor.
1232 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
Another issue is the participants and the staYng levels in the study. The staYng
levels in the study were chosen based on existing and anticipated industry staYng
practices. The operators participating in the study were from a plant where crews
typically consisted of three people. This potential confound, that one of the
experimental staYng levels was identical to the operators’ own working condition,might be expected to result in better performance by the three-man crews; however,
this eVect was not found.
Variables such as motivation, i.e. the Hawthorne eVect, aVect human
performance and are diYcult, if not impossible, to control. Better performance in
the advanced plant than in the conventional plant may be partially explained bymotivation. The conventional plant interface was the participants’ own training
simulator. In contrast, the advanced plant interface was a novel simulator. Crews in
the advanced plant condition may have been more motivated than conventional
plant crews.
EVorts to counteract motivation eVects were taken: modi® cations were made to
the conventional training simulator and operators received training on thediVerences. In both experimental conditions, operators were videotaped and
observed from the gallery. They received considerable attention from experimental
personnel. If operators were more motivated by additional attention, they received
extra attention in both conditions.
The modi® cations to the conventional simulator (i.e. computerized interfacesupport capabilities were not available and steam generator capacity was altered in
the plant model) may have disrupted performance. To minimize this potential eVect,
all crews received training and hands-on simulator practice speci® cally related to
these changes.
DiVerences in training on the two simulator interfaces (2.5 days in the advancedcondition, 2 h in the conventional) may account for a degree of the improved crew
performance in the advanced interface condition. However, given the crews’ months
or years of experience in the conventional simulator, this eVect seems unlikely.
The focus of the study was to obtain realistic, generalizable results. Experimental
control was provided in a carefully considered balance with realistic conditions.
While confounds may have aVected results, eVorts were made to identify and preventthem. By designing scenarios so faults occurred independently of operator actions,
slightly modifying the crews’ own training simulator, providing training on both
interface conditions, and giving all crew types equal attention during the
experimental study, the eVects of potential confounds were minimized.
5. Conclusion
This study was a realistic experiment comparing the eVects of interface design and
staYng levels on crew performance. The ® ndings reveal that interface design and
staYng levels both aVect diVerent aspects of crew performance.
If the interface presents data rather than information (i.e. the conventional typeof plant interface), thereby requiring operators to integrate information, a larger
crew appears more eVective. Essentially, the extra person is available to help gather
information and perform tasks. In an advanced, highly computerized interface,
where information is integrated, an extra person appears to be of little bene® t. In the
advanced plant, integrated information appeared to help the crews understand the
process without requiring additional people. In fact, the larger crews in the advancedcondition had lower situation awareness.
1233Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
In general, the advanced plant interface condition appeared to support better
crew performance in terms of rated team interactions, rated crew performance, and
objective performance.
The concerns identi® ed (Idaszak 1989, O’Hara and Hall 1992, Woods 1984)
about advanced interfaces presenting information in an less than optimally usableformat do not appear to be supported in this study. It should be mentioned that
further research is needed before these issues can be de® nitively addressed, but this
study indicates that advanced technology does, in fact, support crew performance.
However, this improved performance did not appear to be the result of
improvements in situation awareness.Workload is a critical issue in control room design. While performance was
generally better for crews using the advanced plant interface, workload was
signi® cantly higher. This problem was made worse by reducing the staYng level.
Even though no performance decrements were noted, the eVect of maintained high
workload is that performance may drop suddenly (BergstroÈ m 1993). Higher
workload was found in the advanced plant condition, and does not support vendorassumptions that computerized interfaces and plant design features automatically
reduce the operating crew’s workload. Rather, the opposite appears true, and the
increased workload is even more pronounced in smaller crews. Further consideration
should be given to the issue of workload before staYng levels are reduced in
advanced plant control rooms.By using a variety of measures, this study provided a diverse view of crew
performance in realistic process control operating conditions. The results indicate
that interface type and crew staYng levels aVect crew performance in a variety of
ways. Anticipated changes to control room interface design and / or staYng levels
need to be investigated before being implemented.
Acknowledgements
Many people deserve special recognition about the completion of this study and
development of this paper. J. Persensky and Dolores Morisseau, USNRC, provided
funding and guidance. Bruce Hallbert managed the research at the OECD HaldenReactor Project. Mark Green provided guidance and encouragement on this paper.
Marit Larsen assisted with graphics. Ray Saarni, Pekka Pyy and Pekka Sammatti
contributed as process and training experts in the design and running of experiments.
Finally, many thanks to all at the OECD Halden Reactor Project who contributed to
making this project a success.
ReferencesABB ATOM PRELIMINARY SAFETY INFORMATION DOCUMENT 1989, PIUS: The PIUS Arrangement
Principle for a Simpli® ed Passive Advanced Light Water Reactor Plant (VaÈ steraÊ s,Sweden: ABB Atom).
ATOMIC ENERGY of CANADA Ltd, CANDU OPERATIONS 1989, The CANDU 3 TechnicalDescription. Technical report 74-01371-TED-001 , Revd 3, 15 September.
BAKER, D. P. and SALAS, E. 1992, Principles for measuring teamwork skills, Human Factors, 34,469± 475.
BERGSTROÈ M, B. 1993, Task Demand, Workload, and Performance. FOA Report A 50016-5. 2(National Defense Research Establishment Department for Human Studies) (UmeaÊ ,Sweden: FoÈ rsvarets Forskningsanstalt (FOA)).
1234 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
BRANNICK, M. T., PRINCE, A., PRINCE, C. and SALAS, E. 1995, The measurement of teamprocess, Human Factors, 37, 641± 651.
COOK, T. D. and CAMPBELL, D. T. 1979. Quasi-Experimentation: Design and Analysis Issues forField Settings (London: Houghton MiZin).
COURY, B. G. and TERRANOVA, M. 1991, Collaborative decision making in dynamic systems, inProceedings of the Human Factors Society, 31st Annual Meeting, 944± 948.
DECURNEX, C., MOUM, B. R. and FOÈ RDESTROÈ MMEN, N. T. 1996, Integrated InformationOverview Displays. Internal Halden Work Report of the OECD Halden ReactorProject, HWR-451 (Halden, Norway: OECD Halden Reactor Project).
ENDSLEY, M. R. 1987, The application of human factors to the development of expert systemsfor advanced cockpits, in Proceedings of the Human Factors Society, 31st AnnualMeeting, 1388± 1392.
ENDSLEY, M. R. 1988a, Design and evaluation for situation awareness enhancement, inProceedings of the Human Factors Society 32nd Annual Meeting, 97± 101.
ENDSLEY, M. R. 1988b, Situation awareness global assessment technique (SAGAT), inProceedings of the IEEE National Aerospace and Electronics Conference, NAECON(Dayton, Ohio), 789± 795.
ENDSLEY, M. R. 1990, Predictive utility of an objective measure of situation awareness, inProceedings of the Human Factors Society 34th Annual Meeting, 41± 45.
ENDSLEY, M. R. 1995a, Towards a theory of situation awareness in dynamic systems, HumanFactors, 37, 32± 64.
ENDSLEY, M. R. 1995b, Measure of situation awareness in dynamic systems, Human Factors,37, 65± 84.
FOUSHEE, H. C. 1984, Dyads and triads at 35,000 feet: factors aVecting group process andaircrew performance, American Psychologist, 39, 885± 893.
FOUSHEE, H. C. and HELMREICH, R. L. 1988, Group Interaction and Flight Crew Performance,in E. Wiener and D. Nagel (eds), Human Factors in Aviation (New York: AcademicPress).
FOÈ RDESTROÈ MMEN, N. T. and KVALEM, J. 1996, Halden Man ± Machine Laboratory as of 1996.Internal Halden Work Report of the OECD Halden Reactor Project, HWR-476(Halden, Norway: OECD Halden Reactor Project).
GE NUCLEAR ENERGY 1992, SBWR Standard Safety Analysis Report. Report 25A5113 Revd A.GLICKMAN, A. S., ZIMMER, S., MONTERO, R. C., GUERETTE, P. J., CAMPBELL, W. J., MORGAN, B.
B. Jr and SALAS, E. 1987, The Evolution of Teamwork Skills: An Empirical Assessmentwith Implications for Training. Technical report 87-016 (Orlando: OYce of NavalResearch and Human Factors Division, Naval Training Systems Center).
HALLBERT, B. P., SEBOK, A. L., HAUGSET, K., MORISSEAU, D. S. and PERSENSKY, J. J. 1995,Interim Results of the Study of Control Room Crew StaYng for Advanced PassiveReactor Plants, in Proceedings of the 23rd Water Reactor Safety Meeting (Bethesda).
HANSON, D. J., MEYER, O. R., BLACKMAN, H. S., NELSON, W. R. and HALLBERT, B. P. 1987,Evaluation of Operation Safety at Babcock and Wilcox Plants, vol. 1: Results Overview,NUREG/CR-4966 (Washington, DC: US Nuclear Regulatory Commission).
HILL, S. G., IAVECCHIA, H. P., BYERS, J. C., BITTNER, A. C., ZAKLAD, A. L. and CHRIST, R. E.1992, Comparison of four subjective workload rating scales, Human Factors, 34, 429±439.
HOGG, D. N., FOLLESOÈ , K., TORRALBA, B. and VOLDEN, F. S. 1995, Development of a situationawareness measure to evaluate advanced alarm systems in nuclear power plant controlrooms, Ergonomics, 38, 2394± 2413.
HOOPER, K. N., COURY, B. G. and TERRANOVA, M. 1991, Team performance in dynamicsystems, in Proceedings of the Human Factors Society 35th Annual Meeting (SanFrancisco).
HUEY, B. M. and WICKENS, C. D. (eds) 1993, Workload Transition: Implications for Individualand Team Performance (Washington, DC: National Academy Press).
IDASZAK, J. R. 1989, Human operators in automated systems: the impact of activeparticipation and communication, in Proceedings of the Human Factors Society 33rdAnnual Meeting, 778± 782.
1235Team performance in process control
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
KOMAKI, J. L., DESSELLES, M. L. and BOWMAN, E. D. 1989, De® nitely not a breeze: Extendingan operant model of eVective supervision to teams, Journal of Applied Psychology, 74,522± 529.
MONTGOMERY, J., GADDY, C. and TOQUAM, J. 1991, Team interaction skills evaluation criteriafor nuclear power plan control room operators, in Proceedings of the Human FactorsSociety 35th Annual Meeting, 918± 922.
MONTGOMERY, J. C. and HAUTH, J. T. 1992, Team interaction skills evaluation criteria fornuclear power plant control room operators, in Proceedings of the US Nuclear RegulatorCommission Eighteenth Water Reactor Safety Information Meeting (Bethesda).
MORGAN, B. B. Jr, GLICKMAN, A. S., WOODARD, E. A., SALAS, E. and BLAIWES, A. 1986,Measurement of Team Behaviors in a Navy Training Environment. Technical report 86-014 (Orlando: OYce of Naval Research and Human Factors Division, Naval TrainingSystems Center).
MOSIER, K. L. and CHIDESTER, T. R. 1991, Situation assessment and situation awareness in ateam setting, in Y. Queinnec and F. Daniellou (eds), Designing for Everyone (London:Taylor & Francis), 798± 800.
O’HARA, J. M. and HALL, R. E. 1992, Advanced control rooms and crew performance issues:Implications for human reliability, IEEE Transactions on Nuclear Science, 39, 919± 923.
PARASURAMAN, R., MOLLOY, R. and SINGH, I. L. 1993, Performance consequences ofautomation-induced c̀omplacency’ , International Journal of Aviation Psychology, 3,1± 23.
REGAL, D. M., ROGERS, W. H. and BOUCEK, G. P. Jr 1988, Situational awareness in thecommercial ¯ ight deck: De® nition, measurement, and enhancement, SAE Technical PaperSeries: Aerospace Technological Conference and Exposition (Anaheim, California), 15.
ROTH, E. M., LIN, L., THOMAS, V. M., KERCH, S., KENNEY, S. J. and SUGIBAYASHI, N. 1998,Supporting situation awareness of individuals and teams using group view displays, inProceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, 244±248.
ROTH, E. M., MUMAW, R. J. and LEWIS, P. M. 1994, An Empirical Investigation of OperatorPerformance in Cognitively Demanding Simulated Emergencies. NUREG/CR-6208(Washington, DC: US Nuclear Regulatory Commission).
ROTH, E. M., MUMAW, R. J. and STUBLER, W. F. 1993, Human factors evaluation issues foradvanced control rooms: a research agenda, IEEE Conference Proceedings, 254± 265.
SCHRYVER, J. C. 1994, Experimental validation of navigation workload metrics, in Proceedingsof the Human Factors Society 38th Annual Meeting, 341 ± 344.
SEBOK, A. L., HALLBERT, B. P. and MORISSEAU, D. S. 1996, Crew performance issues in nuclearpower plant process disturbances, in Proceedings of the 1996 American Nuclear SocietyInternational Topical Meeting on Nuclear Plant Instrumentation, Control, and Human ±Machine Interface Technologies (State College, PA), 645651.
STUBLER, W. F. and O’HARA, J. M. 1996, Group-View Displays: Functional Characteristics andReview Criteria. Technical report E2090-T-4-4-12/94 (Upton: Brookhaven NationalLaboratory).
WESTINGHOUSE ELECTRIC CORP. 1992, AP600 Standard Safety Analysis Report, revd 0,Simpli® ed Passive Advanced Light Water Reactor Plant Program Report DE-AC03-90SF18495. Prepared for the US Department of Energy, San Francisco OperationsOYce.
WINER, B. J. 1971, Statistical Principles in Experimental Design, 2nd edn (New York:McGraw-Hill).
WOODS, D. D. 1984, Visual momentum: a concept to improve the cognitive coupling of personand computer, International Journal of Man ± Machine Studies, 21, 229± 244.
1236 A. Sebok
Dow
nloa
ded
by [
Uni
vers
ity o
f N
orth
Dak
ota]
at 0
1:38
03
Sept
embe
r 20
14
