Improving Cockpit Task Management Performance:

Improving Cockpit Task Management Performance:

Aviate

Navigate

Communicate

Manage Systems

The AgendaManagerTraining Pilots to Prioritize Tasks

Aviate

Navigate

Communicate

Manage Systems

CTM

2

Observation: Cockpit Task Management Errors

• Cockpit (flight deck) is a multitask environment– aviate– navigate– communicate– manage systems

• Results of distraction, preoccupation– Everglades L-1011 accident– many incidents

• Hypotheses: – flightcrew must manage as well as perform tasks:

Cockpit Task Management (CTM) – CTM is a significant factor in flight safety

Aviate

Navigate

Communicate

Manage Systems

CTM

3

Preliminary Normative Theory of CTM

• initiate tasks to achieve goals

• assess status of all tasks

• terminate completed and ‘obsolete’ tasks

• prioritize remaining tasks based on– importance:

• aviate

• navigate

• communicate

• manage systems

– urgency

– other factors (?)

• allocate resources (attend) to tasks in order of priority

Aviate

Navigate

Communicate

Manage Systems

CTM

4

Cockpit Task Management Research

• CTM Errors in Aircraft Accidents (1991)– 80 CTM errors in 76 (23%) of 324 accidents

• CTM Errors in Critical, In-Flight Incidents (1993)– 349 CTM errors in 231 (49%) of 470 incident reports

• Part-Task Flight Simulator Study (1996)– CTM error rate increases with workload

• ASRS Study of CTM and Automation (1998)– Task prioritization error rate higher in advanced technology reports

• Findings:– CTM is a significant factor in flight safety

– CTM can potentially be improved

Aviate

Navigate

Communicate

Manage Systems

CTM

Improving CTM Through Technology:

The AgendaManager

5

Aviate

Navigate

Communicate

Manage Systems

CTM

6

Statement of Needs and Requirements Definition

• CTM aid shall– maintain a current model of aircraft state and current

cockpit tasks,

– monitor task state and status,

– compute task priority,

– remind the flightcrew of all tasks that should be in progress, and

– suggest that the flightcrew attend to tasks that do not show satisfactory progress.

– leave the pilot in control

Aviate

Navigate

Communicate

Manage Systems

CTM

7

System Analysis

• Generic, twin-engine transport aircraft– major subsystems

• power plant

• fuel system

• electrical system

• hydraulic system

• adverse weather system

• autoflight system

• flight management system.

– state variables of importance to pilot

specifications for simulator

Aviate

Navigate

Communicate

Manage Systems

CTM

8

Basic and Detailed Design of

The AgendaManager• Object-Oriented Design

– things & activities from IDEF0 models objects

• Multi-Agent Approach– AMgt functions are complex, cognitive functions AI– AMgt is complex interplay of many entities DAI

• System Agents• Actor Agents• Goal Agents• Function Agents• Agenda Agent• Agenda Manager Interface

• Display Design– general display design guidelines alternative display designs– consistency with EICAS final display design

Autoflight

L Engine

Hyd System

Fuel System

Aircraft

Pilot

Aircraft Agent

Autoflight Agent

L Engine Agent

Hyd System Agent

Fuel System Agent

Flightcrew Agent

Simulator

AMgr display

Verbex ASR

AgendaManager

descend to 9,000 ft G & F Agents

descend to 8,000 ft G & F Agents

extinguish L ENGINE FIRE G & F Agents

restore C HYD PRESS G & F Agents

correct FUEL BALANCE G & F Agents

Goal & Function AgentsSystem AgentsSystem Models

reduce to 240 kt G & F Agents

maintain 070 deg G & F Agents

maintain 070 deg G & F Agents

reduce to 240 kt Goal & Function Agents

information flow

unsatisfactory functionsconflicting goals

satisfactory functions

AMgr Architecture and Function

Aviate

Navigate

Communicate

Manage Systems

CTM

10

Simulator(with EICAS)

Aviate

Navigate

Communicate

Manage Systems

CTM

11

AMgr Display(replaced EICAS)

Aviate

Navigate

Communicate

Manage Systems

CTM

12

AMgr Operation

• simulator runs• pilot declares goals via ATC acknowledgements• System & Actor Agents instantiate Goal Agents• Goal Agents watch for goal conflicts• Function Agents assess function status• AgendaManager informs pilot via display

extinguish L engine fire not OK -> continuing

slow to 240 kt fast -> acceleratingmaintain 070 degdescend to 7,000 ft A/F alt goal conflict

correct fuel balance L heavy -> increasing

AgendaManager Display Design

extinguish L engine fire not OK -> continuing

slow to 240 kt fast -> acceleratingmaintain 070 degdescend to 7,000 ft A/F alt goal conflict

correct fuel balance L heavy -> increasing

extremely important, urgent goals

(highest priority) trend info

aviate goals(high priority)

system goals(lower priority)

gray = OKamber = not OKred = important/urgent not OK

maintain 280 ktmaintain 120 degmaintain15,000 ft

Initial Conditions:altitude = 15,000 ftheading = 120 degspeed = 280 ktall systems normal

maintain 280 ktmaintain 120 degdescend to 11,000 ft high -> descending

ATC: “... descend and maintain 11,000 ft”pilot: “Roger, “... descend and maintain 11,000 ft”

sets A/F altitude to 11,000 ftdescent begins

maintain 280 ktturn L to 070 deg right of -> turning Lmaintain 11,000 ft

ATC: “... turn left heading 070”pilot: “Roger, “... turn left heading 070”

begins turnlevels off at 11,000 ft

maintain 280 ktmaintain 070 degmaintain 11,000 ft

correct fuel balance L heavy -> unbalancing

pilot: rolls out on 070 degAMgr:detects fuel imbalance & displays it

slow to 240 kt fast -> slowingmaintain 070 degdescend to 9,000 ft high -> descending

correct fuel balance L heavy -> balancing

pilot: begins fuel crossfeedATC: “... descend and maintain 9,000 ft; reduce

speed to 240 kt”pilot: “Roger ... descend and maintain 9,000 ft;

reduce speed to 240 kt”sets altitude to 9,000 ft, descent beginsreduces throttles, aircraft slows

extinguish L engine fire not OK -> continuing

slow to 240 kt fast -> acceleratingmaintain 070 degdescend to 7,000 ft A/F alt goal conflict

correct fuel balance L heavy -> balancing

AMgr: detects left engine firepilot: “... we have a problem ...”ATC: “... descend and maintain 7,000 ft”pilot: “Roger ... descend and maintain 7,000 ft”

mis-sets altitude to 6,000 ftspeed increases

maintain 240 ktmaintain 070 degmaintain 7,000 ft

correct fuel balance R heavy -> unbalancing

fire outspeed controlledpilot: sets A/F to 7,000 ft

forgets to secure crossfeed when fuel balanced

Aviate

Navigate

Communicate

Manage Systems

CTM

22

Test and Evaluation (1)

• Objective: compare AMgt performance (AMgr vs EICAS)

• Apparatus– flight simulator

– AMgr

• Subjects: 8 line pilots

• Scenarios:– EUG to PDX

– PDX to Eugene

• Primary factor: monitoring and alerting condition– AMgr

– EICAS

Aviate

Navigate

Communicate

Manage Systems

CTM

23

Test and Evaluation (2)

• General Procedure– subject introduction– automatic Speech Recognition system training– flight training (using MCP)– subsystem training (fault correction)

– EICAS/AMgr training

• Trials– Scenario 1 (EICAS/AMgr)

• experimenter/ATC controller gives clearances, induces faults, induces goal conflicts

• subject acknowledges clearances, flies simulator, corrects faults, detects and resolves goal conflicts

– Scenario 2 (AMgr/EICAS)

Aviate

Navigate

Communicate

Manage Systems

CTM

24

Response variable AMgr EICAS sig. levelwithin subsystem correct prioritization 100% 100% NS

susbsystem fault correction time (sec) 19.5 19.6 NS

autoflight system programming time (sec) 7.0 5.9 NS

goal conflicts corrected percentage 100% 70% 0.10

goal conflict resolution time (sec) 34.7 53.6 0.10

subsystem/aviate correct prioritization 72% 46% 0.05

average number of unsatisfactory functions 0.64 0.85 0.05

percentage of time all functions satisfactory 65% 52% 0.05

mean subject effectiveness rating (-5 to 5) 4.8 2.5 0.05

Evaluation Results

Aviate

Navigate

Communicate

Manage Systems

CTM

25

Conclusions

• CTM is a significant factor in flight safety.• CTM can be facilitated (e.g., AMgr).• Future success of knowledge-based avionics depends

on a systematic approach to development:– systematic identification of problems, needs, and

opportunities

– appropriate application of appropriate technology

– evaluation of systems based on operationally relevant performance measures

Aviate

Navigate

Communicate

Manage Systems

CTM

Improving CTM Through Training:

Training Pilots to Prioritize Tasks

26

Aviate

Navigate

Communicate

Manage Systems

CTM

ResearchMotivation and Objective

• Is task prioritization trainable?• Evidence suggests that voluntary control of

attention is a trainable skill– e.g., Gopher (1992)

• Objective– Develop and evaluate a CTM training program to

improve task prioritization performance.

Aviate

Navigate

Communicate

Manage Systems

CTMMethodology

• Participants– 12 General Aviation pilots, IFR rated, with at least 100 hrs “pilot-

in-command” total time.

– Recruited through flyers and word of mouth

– Oregon State (Corvallis, Albany, Salem, Eugene, Portland)

• Apparatus: Microsoft Flight Simulator 2000– 3 monitors, Flight Yoke, Throttles, and Rudder Pedals

– IFR conditions

– Two flight scenarios

Aviate

Navigate

Communicate

Manage Systems

CTM

Lab Setup

Aviate

Navigate

Communicate

Manage Systems

CTM

Participant Display(C-182RG)

Aviate

Navigate

Communicate

Manage Systems

CTM

Experimenter’s Display

Aviate

Navigate

Communicate

Manage Systems

CTM

Experimental Groups

• Control Group: No Training• Descriptive Group: CTM lecture

• Multi-tasking• Attention• CTM• Task Prioritization errors• Accident/Incident examples• What to be aware of.

• Prescriptive Group:– CTM lecture – “APE” procedure

Aviate

Navigate

Communicate

Manage Systems

CTM

APE:Assess Prioritize Execute

• Let the APE help you– Assess the situation:

• aircraft systems, environment, tasks, procedures• “What’s going on?” “What should I be doing?”

– Prioritize your tasks:1. Aviate: “Is my aircraft in control?”2. Navigate: “Do I know where I am and where I’m going?”3. Communicate: “Have I communicated or received important information?”4. Manage systems: “Are my systems okay?”

– Execute the high priority tasks Now.

• Invoke the APE frequently.• Think out loud.

A P E

Aviate

Navigate

Communicate

Manage Systems

CTM

Experimental Procedure

1. Initial briefing, informed consent

2. Initial 30-minute simulator training

3. Pre-training flight

4. CTM training (break for control group)

5. Additional 30-minute simulator training

6. Post-training flight (different scenario)

7. Post-experiment questionnaire

Aviate

Navigate

Communicate

Manage Systems

CTM

Dependent Measures

• Task prioritization error rate– 19 Task prioritization challenges, e.g.

• clearance near end of climb

• “bust” altitude? (+/- 200 ft)

• Prospective memory recall rate– 5 Memory recall challenges (prospective memory), e.g.,

• “report crossing SHONE [intersection]”

• remember to report?

Aviate

Navigate

Communicate

Manage Systems

CTM

Data Collection

• Flight Data Recorder• Videotape• Observation• Data reduction to:

– task prioritization error rate

– prospective memory recall rate

Aviate

Navigate

Communicate

Manage Systems

CTM

Results: ANOVA(task prioritization error rate)

Effect df SS df MS F-ratio p-value

Group 2 .1914125 9 .0799194 2.395 .147

Flight 1 .2053500 9 .0088806 23.123 .001

Group x Flight 2 .0429125 9 .0088806 4.832 .038

Aviate

Navigate

Communicate

Manage Systems

CTM

Interaction Plot(task prioritization error rate)

Interaction Plot

Flight

Err

or R

ate

GroupControlDescriptivePrescriptive

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Pre Training Post Training

Control

Descriptive

Prescriptive

Aviate

Navigate

Communicate

Manage Systems

CTM

Results: ANOVA(prospective memory recall rate)

Effect df SS df MS F-ratio p-value

Group 2 .017 9 .028 .603 .568

Flight 1 .074 9 .034 2.181 .174

Group x Flight 2 .171 9 .034 5.055 .034

Aviate

Navigate

Communicate

Manage Systems

CTM

Interaction Plot(prospective memory recall rate)

Interaction Plot

Flight

Mem

ory

Tas

ks

GroupControlDescriptivePrescriptive

0.5

0.6

0.7

0.8

0.9

1

Pre Training Post Training

Prescriptive

Descriptive

Control

Aviate

Navigate

Communicate

Manage Systems

CTM

Paired t-tests

• Prescriptive training group improved• Task prioritization error rate

• Prospective memory recall rate

• Descriptive training group improved• Task prioritization error rate

• Control group did not significantly improve

Aviate

Navigate

Communicate

Manage Systems

CTM

Discussion

• Task Prioritization Error rate– Reduced, perhaps, due to (Prescriptive) CTM training.– Significant interaction and post-hoc tests support that

hypothesis.

• Prospective Memory Recall rate– Increased, perhaps, due to (Descriptive & Prescriptive) CTM

training.– Significant interaction and post-hoc tests support that

hypothesis.

Aviate

Navigate

Communicate

Manage Systems

CTM

Possible Interpretations

• Results may have two interpretations:1. CTM training did improve task prioritization performance.

2. CTM training did not improve task prioritization.• Floor effect

• MSFS experience

• Age

• Research favors first interpretation• ANOVA results

• t-tests

• Potential for better control group performance was there.

• Additional tests

Aviate

Navigate

Communicate

Manage Systems

CTM

Final Comments

• CTM performance significant to flight safety• Results are encouraging• Evidence suggests that task prioritization is a

trainable skill• Follow-up experiment underway to resolve

ambiguities• If successful, would provide evidence that CTM

training can reduce risk of CTM errors and subsequent accidents

Aviate

Navigate

Communicate

Manage Systems

CTM

The AMgr: a KBS 45

The Cockpit Task Management Website

http://flightdeck.ie.orst.edu/CTM/