NEUROPHYSIOLOGICAL WORKLOAD ASSESSMENT IN FLIGHT Tom Schnell, Mike Keller and Pieter Poolman, Operator Performance Laboratory (OPL),

Iowa City, Iowa

Abstract Over the past few years, we developed a

neurocognitive assessment system called the Cognitive Avionics Tool Set (CATS). This system has been used in fixed and rotary wing aircraft and in automobiles to assess operator workload using physiological and neurocognitive markers. Sensors that we used in the past include dense array electroencephalogram (EEG), electrocardiogram (ECG), galvanic skin response (GSR), pulse oximetry (sPO2), respiration (amplitude and frequency), and non-contact measures such as facial feature point location, facial temperature differences, and eye tracking. We recently integrated a new sensor called the PhotrodeTM manufactured by SRICO. The Photrode is an optical device that detects electrical fields. The device operates by sending a beam of light through a Mach-Zehnder Interferometer (MZI). An electric field modulates the light in a way that can be detected by an optical receiver. Because bio-electrical fields travel through clothing, the Photrode can detect ECG activity remotely, eliminating the need for wires being directly attached to the individual. The Photrode is capable of detecting EEG activity remotely through clothing. This paper provides an overview of CATS.

Introduction Considerable progress has been made in recent

years in the area of physiological based pilot state characterization. Based on a multi-year NASA contract, the research group at the Operator Performance Laboratory (OPL) has developed a framework called the Cognitive Assessment Tool Set (CATS) and tested it extensively on their Computerized Airborne Research Platform (CARP) research aircraft. This multi-sensory physiological based tool set (Schnell et al., 2007b) enables the connectivity of multiple sensors into one unified framework, it synchronizes data, removes artifacts, and discriminates operator state using a neural network classification system.

Schnell et al. (2006) discussed results of a fixed wing study that was conducted to determine neural markers for cognitive, perceptual, and motor loading in several flight maneuvers. The maneuvers included tasks that were specially designed to isolate the effects of muscle movement artifacts on EEG in support of the CATS automatic artifact removal algorithms. Schnell et al. concluded that with proper artifact removal, there were strong neural markers of cognitive and motor loading, including EEG alpha, beta, and theta power. They also indicated the need for multi-sensory technology to enhance the reliability of the pilot state characterization. Schnell et al. (2007b) reported on the results of a rotary wing flight test of CATS that was conducted on a Bell 412 research helicopter. Several easy and difficult flight tasks were used to elicit contrasting neural responses. Data was presented for two contrasting participants, an experienced high time military pilot and a non-pilot researcher with moderate simulator experience on the Bell 412. The CATS tool was used to explore the clinical EEG bands for each participant and each task across the spatial extent of the 128 electrodes. Of particular interest was the determination of specific regions on the scalp that conveyed significant neural markers of workload and stress. Both participants showed the lowest beta/alpha ratio in occipital EEG during the rest task. Consistent with the difficulty of the task, the pilots exhibited the highest beta/alpha ratio during the most difficult flight maneuvers. The data indicated that it is possible to differentiate between levels of cognitive performance in experienced and novice pilots on the basis of EEG and flight technical data in a technically challenging environment. CATS is a tested and fully integrated cognitive and physiological status monitoring (CPSM) system that features an architecture to synchronize system and mission state data with a multitude of physiological sensors such as dense array electroencephalogram (EEG), electrocardiogram (EKG), electromyogram (EMG), electrooculogram (EOG), galvanic skin response (GSR), pulse oximetry (SpO2), pulse wave (blood

978-1-4244-2208-1/08/$25.00 ©2008 IEEE. 4.B.2-1

pressure), respiration (amplitude and frequency), and non-contact measures such as facial feature point location, facial temperature differences, and eye tracking. CATS is, to our knowledge, the most complete, flexible, and mature architecture that can accomplish multi-sensory cognitive CPSM.

Human Aware Avionics

Operator State Classification and Feedback (OSCAF)

The goal of our work is to design an avionics human-machine interface that is aware of the state of the pilot. Through such an interface, human aware avionics would be able to detect when the crew is not operating at the desirable cognitive level and appropriate countermeasures could be taken. For example, it would be possible to ensure that the crew is not asleep, incapacitated, or overloaded. There are numerous reports of crews who after long crew duty days have become drowsy or even fell asleep. A human-aware avionics interface can detect drowsy states with relative ease and high reliability. Although rare, crews can also become incapacitated due to hypoxia. For example, Helios airlines Flight CY522 crashed on 14 August 2005 at 12:04 EEST into a mountain north of Marathon and Varnavas, Greece [1] . The cause of the crash was attributed to lack of pressurization and subsequent hypoxia. The aircraft continued on automation until it reached a fix above its destination and eventually ran out of fuel. Most likely due to time-pressure, the pressurization panel was not properly set up by the first officer prior to take-off. A short time after take-off the cabin altitude warning was activated as a result of pressurization. However, the crew misidentified the warning as a take-off configuration warning. The warning was silenced by the crew with a switch on the overhead panel.

When the aircraft climbed above 14,000 ft (4,300 m) the oxygen masks in the cabin automatically deployed. An Oxy ON warning light

on the overhead panel in the cabin illuminated when this happened. Minutes later a master caution warning light activated, indicating an abnormal situation in a system. This was misinterpreted by the crew that systems were overheating. Quite clearly, this crew ignored the standard cabin altitude warnings and a human-aware avionics interface would have been able to detect the onset of hypoxia and take appropriate countermeasures.

Our Operator State Classification and Feedback (OSCAF) system is shown in Figure 1. At this time, we have not implemented all parts of this system. We focused our initial effort on the development of the Human Cognitive and Physiological system. We refer to this subsystem as the Cognitive Avionics Tool Set (CATS) which has been earlier described by Schnell et al. [2-6].

The OSCAF system shown in Figure 1 represents a real-time closed-loop monitoring architecture that assimilates neural, physiological, and aircraft state data to determine operator state. The loop is then closed through a set of mitigations that warn the pilot of undesirable states and fosters desirable states. Aircraft state data is sent to a Pilot Controller Model (Figure 1) that generates expected flight control output values. This controller based model can predict expected flight control inputs from aircraft state. By comparison of expected and actual flight control inputs, it is possible to assess if the pilot is responsive. If, for example, a perturbation such as a wind gust causes the aircraft to bank, we would expect the pilot to generate a compensatory aileron control input. Goal states for the human operator, the aircraft Automation, the Pilot Controller Model, and the Mitigation Trigger are represented in the form of flight plans, approach procedures, aircraft limitation data (V-speeds, etc.) and phases of flight. State data from the various sources is concentrated in CATS where data synchronization and data artifact removal takes place.

4.B.2-2

Respiration belt placed under thesternum notch

Human Cognitive and Physiological system:EEG, EMG, Eyetracking, Facial Features, etc.

Aircraft: Pitch, Roll, Yaw, Speed. Altitude, Rates, Position, Control Inputs, etc.

Pilot Controller Model:Input Aircraft States and Mission State, Output Flight Control Activity

Aircraft State

Vector

State Requirements:Instrument approaches, MEAs, DH, MDAs, V-Speeds, etc.

Automation: Input Aircraft States, State Requirements, User Inputs, Output Flight Control Activity

Flight Plan, Procedures,

Preselects, etc.

Aircraft State Vector

Flight Control Displacement

Visual, Auditory, Haptic,

Vestibular, etc. Feedback

Actual Flight Control Inputs

Expected Flight Control

Inputs

Goal States such as speed, altitude, attitude, location

Interaction with

Automation

Physiological Signals

Aircraft State Vector

Data Synchronization, Fusion, Cleaning, Filtering, Power Spectra

Automation StateUnified

State Data

Auditory Mitigation: Verbal Warnings (Jenny Device), Warning and State Sounds

Frequency (Hz)

Ampl

itude

(db)

Adjustable center frequency

Tactile Mitigation: Tactile Suit or tactile seat

Visual Mitigation: Background ADI, SVS Turns on, Cautions and Warnings including Biofeedback

Wor

kloa

d

Con

flict

Situ

atio

nal

Awar

enes

s

Operator State Classification Model:Input Unified State Data, Output: Workload, Conflict, Situation Awareness

Unified State Data

Verb

al o

r S

patia

lMitigation Trigger:Determines if Countermeasures need to be activated and how

Figure 1. Functional Diagram of the Proposed Operator State Classification and Feedback System

Operator State Classification module inside of CATS

Flight technical and mission technical measures are as important a part of operator state characterization as physiological measures. Flight technical measures indicate a pilot’s ability to generate flight trajectories that are commensurate to

those prescribed in the mission state requirements module. Continuous measures per mission task element (MTE) will be computed into an overall performance score using Equation 1.

4.B.2-3

∑=⋅=

n

i iiiCM MOEFWP1

)( (1)

where Wi is the ith weight of n weights of importance and MOEi is the ith of n continuous measures of effectiveness. Also, . Each ith continuous measure of effectiveness will be subjected to an individual cost function Fi that returns 1 for perfect performance, intermediate values for intermediate performance, and 0 for performance outside the set tolerance as shown

in

∑ =1W

Figure 2. These cost functions are tailored to thspecific MOE and there are cost functions that represent “lower-is-better”, “nominal-is-best”, or “higher-is-better” performance expectations. The algorithm could include a “knock-out” criteria, where a fail score in a cost function could force the entire performance score to fail. The weights of importance will differ for different mission task elements (MTEs) to account for the different flight strategies that are applied. The continuous mission performance score PCM are updated on a rolling RMS window of 5 seconds.

e

1.0

0.0

Best Upper Tolerance

Cos

t Fu

nctio

n F

Lower is Better Measure of Effectiveness

1.0

0.0

Best Upper Tolerance

Lower Tolerance

Cos

t Fu

nctio

n F

Nominal Best Measure of Effectiveness

1.0

0.0

BestLower Tolerance

Cos

t Fu

nctio

n F

Higher is Better Measure of Effectiveness

Figure 2. Cost Function to Normalize Continuous Performance Scores

We also use discrete mission performance metrics PDM to account for occurrence of discrete tasks such as activating switches. These discrete scores will range from 0 to 1 as well and cost functions will be applied as well. For example, if a pilot forgets to lower the landing gear, or the flaps are set to the wrong configuration, the score will be 0. If a pilot forgets to activate these switches but discovers the lapse with sufficient time for recovery, the cost function will weigh the score with the lapse time from when the switch should have been activated to when it was actually activated. Discrete metrics are collected as shown in Equation 2.

∑ =⋅=

m

j jjjDM MOEFWP1

)(

DMkCMT PPP

(2)

Finally, the continuous and discrete metrics are multiplied to obtain a total score PT as shown in Equation 3.

= + (3)

The total score PT will provide CATS with a moving window gauge of technical performance and it will be possible to compare the pilot’s cognitive loading and resulting flight technical performance in real-time. Examples of flight technical errors that may be useful in describing operator state include those shown in Table 1.

A unified state data vector that contains information on the aircraft and pilot is sent to the Operator State Classification model in CATS.

4.B.2-4

Table 1. Overview of Flight Technical Measures of Effectiveness (MOEs)

MOE Number

MOE Name Description

1 Cross Track Error (XTE) Lateral deviation from the ideal flight path. Smaller is better.

2 Vertical Track Error (VTE) Vertical deviation from the ideal flight path. Smaller is better.

3 Speed Error (SE) Deviation from commanded speed. Smaller is better.

4. Altitude Error (AE) Deviation from commanded altitude. Smaller is better.

5 Heading Error (HE) Deviation from commanded heading. Smaller is better

6 Heading Capture Error (HCE) Deviation from specified heading as aircraft rolls to wings level. Smaller is better.

7 Aileron Input Amplitude (AIA) Analysis for lateral control input amplitude. Smaller is better but subject to adequate position errors.

Elevator Input Amplitude (EIA) Analysis for vertical control input amplitude. Smaller is better but subject to adequate position errors.

8 Aileron Input Frequency (AIF) Analysis for lateral control input frequency. Smaller is better

9 Elevator Input Frequency (EIF) Analysis for vertical control input frequency. Smaller is better

10 Roll Reversals (RR) The number of times the aircraft crosses the zero axis in roll during flight.

The neural network at the heart of the classifier accepts multiple input vectors from which it classifies operator state as shown in Figure 3. The physiological state vector is one such inputs. The physiological state vector includes all of the data components derived from the subject’s body including neural activity, heart rate variability, and facial thermography, amongst others. New sensors such as the PhotrodeTM or a pulse wave sensor can be easily integrated into physiological vector. Further, sensors can easily be removed from the input vector if they prove unnecessary or impractical for a particular application or environment. New analysis techniques can also be

easily added or removed from the physiological input vector. The neural network also accepts equally important vectors from the aircraft system, which includes both flight controls, system switches (e.g. weapons master, laser arm, mission computer keypad, etc.) and aircraft state (e.g., airspeed, pitch, roll). Control inputs give information about pilot workload, pilot intent and situation awareness. The aircraft state vector contains all of the information about the aircraft state and can be used to assess flight technical performance. The classification system in CATS uses different data processing steps for different physiological measures as indicated in Figure 3.

4.B.2-5

Figure 3. Operator State Classification Data Flow

No single vector alone provides the complete picture. A true understanding of the situation requires an understanding of all of the input vectors. It is only from this composite picture that the classifier is able to intelligently assess operator state.

Before it can classify new data, the neural network must be trained with tagged sample data. We will employ multiple training strategies. First, we will train against self-reported workload scores including scoring techniques such as the NASA TLX. Self reported scores are good in that they reflect (or at least attempt to reflect) the individual difficulty of each task for a specific individual. However, the value of the self-reported workload score depends upon the accuracy with which subjects self-report. It is critical that subjects be sufficiently briefed on the workload ratings technique prior to use to ensure subjects provide accurate ratings.

A second training mechanism uses expert ratings of task difficulty. A panel of experts is used

to rate each task or maneuver a priori according to a pre-defined scale. The expert ratings have the benefit of eliminating variance between subjects. However, some tasks may be fundamentally easier for some pilots to complete than for other pilots, making the ratings possibly ambiguous.

Next, we train the network to recognize and differentiate the signatures of trained experts from that of novices. In addition to multi-layer perceptron artificial neural networks, we are also investigating the usability of unsupervised self organizing map (SOM). These maps automatically cluster similar patterns without any external tagging or training. SOMs are designed to reveal the underlying dimensionality of the incoming data, even if the dimensionality is not known a priori. The SOMs are useful because they may demonstrate what dimensionalities of workload exist in the aviation domain. They may also be used to compare the relative difficulty of a new task to others that have already been tested.

4.B.2-6

After classification, the OSCAF architecture attempts to close the loop by issuing mitigating warnings to the pilot. The Mitigation Trigger (see Figure 1) takes into account the unified state data and the operator state variables coming fromthe Operator State Classification Model to makedetermination as to the type, magnitude, and directionality of the mitigation systems that need to be triggered. Of course, the mitigation systems are expected to affect the operator in a fashion that is productive in resolving the issue sensed by the Operator State Classification Model. If operator vital signs or aircraft state data would indicate that the problem gets worse, it may be necessary to suppress the mitigation systems to avoid feed-forward runaway cueing.

a

Real-Time Architecture We employ a sophisticated, flight-proven

system architecture to manage data flow both on our target platforms known as the Common

Asynchronous Receive/Transfer (CART) protocol. The CART protocol enables many data producers and data consumers to communicate without expensive configuration. Data is transmitted through standardized “buses”. Buses are self-contained carriers of information. Servers can send buses and clients can received buses without the rest of the architecture knowing anything about the bus. This makes adding or removing buses of information very easy.

Data synchronization occurs automatically and in real-time. The architecture can handle buses operating at different data rates. When queried, the database returns the most relevant frame from each of the buses, see Figure 4 and Figure 5. The CART protocol supports a number of transports, including multicast UDP, TCP, High Level Architecture (HLA), direct FireWire (IEEE 1394) and via shared memory. The protocol has been implemented on multiple computer platforms including WIN32, Linux and OS-X.

Figure 4. Example of Buses Operating at Different Rates

Figure 5. Example Query for Buses Operating at Different Rates

The CART has built-in recording capabilities that log to a index-able database. CART data can be analyzed post hoc using the Cognitive Avionics Toolset (CATS) or in real-time using the CATS-RT visualization environments. The protocol natively handles buses that operate and different rates. The protocol also handles asynchronous “bursty” sources that are typical in sources with high update rates.

Data is collected from a wide variety of sensors. These sensors include the physiological sensors that monitor the pilot (see Figure 3). These sensors are symbolized with dark blue boxes in Figure 3. The raw buses are then time synchronized. Many of the signals are then filter(light blue boxes in

ed

nals are Figure 3). The nature of the

filter depends on the source signal. Most sigthen subjected to a digital processing algorithm (purple boxes in Figure 3). Some signals such as

4.B.2-7

ECG go through multiple layers of filtration anddigital signal processing. Finally, metrics are computed (orange boxes in

c may le

Figure 3). A metribe derived from signals originating from multipsources. These metrics are then fed into the neural network. The neural network output is fed through a stateful state discriminator. The discriminator provides the operator state classification.

Sensors and Integration The CATS architecture can accommodate any

number of sensors and data sources. Our current flight package has been documented in [2-6]. We use a dense array Electro-encephalogram (EEG) system with 128 sensors to acquire neural activity. A three channel electrocardiogram (ECG) sensor is used to measure cardiac electrical activity (Figure 6). The ECG strongly reflects sympathetic and parasympathetic activity in the autonomous

nervous system. Heart rate alone is affected by several factors, making it less valuable for assessing autonomic activity. However, heart rate variability (HRV), or the change in heart rate from beat to beat, provides strong indication of autonomic activity. Several analytical methods have been applied to the analysis of HRV including the discrete Fourier transform, the discrete wavelet transform, Poincare analysis. The Puls Atem Quotient which compares the heart rate to breathing rate also provides an indication of stress. To that end, we measure respiration amplitude and frequency with an elastic band around the rib cage or with a sensor in the pilot’s oxygen mask (for military jet applications). We also deploy a pulse-oximetry sensor to measure blood oxygenation, an eye tracker to determine visual attention and visual scanning activity, and a thermal camera to obtain real-time readings of facial temperature.

EEG ECG and HRV Respiration Eye Tracking and Facial Thermography

Figure 6. Basic Sensor Modalities Used in CATS

One important aspect of a human-aware interface is the deployment of sensors on the pilot. For many of our flight tests in the past, we used clinical sensor equipment that was donned on the pilot before flight. Many of these sensors are rather intrusive and we realize that for a production system the deployment of sensors has to be very simple and fast. This requirement gave rise to the Cognitive Pilot Helmet (CPH) concept. With funding from Rockwell Collins (RCI), OPL researchers have recently built the first prototype of an aviator helmet outfitted with neurophysiological sensors for use with simulators and real aircraft, like the Joint Strike Fighter – see Figure 7. We

designed an inner shell liner that precisely conforms to the shape of the pilot’s head. Into that liner, we embedded 32 ActiCap active EEEG electrodes, a pulse-oximeter, and eye tracking camera (see Figure 8). This shell liner is completely rigid with respect to the pilot’s head, thus guaranteeidentical sensor placement each time the helmet is donned. The shell liner fits relatively loosely into the aviator helmet to allow for vibrations to be absorbed by the helmet rather than by the shell liner. Using the helmet as a sensor farm for collecting physiological and neurophysiological signals is an exciting new concept and the precise fit will guarantee a good test-retest quality.

ing

4.B.2-8

Figure7. Sensor Layer and Helmet Shell (Left), with Connectivity Diagram (Right) of the Cognitive Pilot

Helmet

Figure 8. Eye Tracking Camera

CATS User Interface The CATS Graphical User Interface (GUI) has

been designed with regard to real-time and post-run classification of operator state in a flight test environment. The real-time system can be operated in an unattended mode but it has provisions for user interaction to monitor signal quality. There are many separate screens in CATS that deal with data selection, data query, integrity monitoring, data analysis, state classification, etc. Only small subsets of all screens are shown in this paper.

The CATS data collection architecture has been set up to generate a synchronized single database of the data sources that produce data at various sampling rates on the aircraft. Data can be down selected for analysis using a query method. For example, a query could be generated to analyze

data for Pilot 1 flying Maneuver 1. Or a query could be generated to analyze data in all pilots and all left banking turns exceeding 80 degrees angle of bank and an acceleration of more than 2 g. The selection of data files and queries can be performed from the CATS main screen (see Figure 9). The user can select which data files from which sorties should be analyzed. Through the Query button, the user can enter a data reduction query using data tags that describe events or states. Data tags can be automatically generated using threshold criteria or they can be user entered during data collection time.

From the main screen, the user can reach screens that contain detailed data about the different sensors such as EEG, thermo camera, eye tracker, etc. Other pages include information on signal integrity monitoring or operator state classification.

CATS also offers a graphical query tool called the Earth Viewer as shown in Figure 10. Using this tool, the user can select a portion of interest of the flight using a graphical tool with photorealistic texture imagery. This is an excellent tool to extract extraneous data that is often collected during repositioning of the aircraft for the subsequent maneuver. Figure 11 shows an example of a waveform viewing page, in this case for the EEG signals.

4.B.2-9

Figure 7. CATS Main Screen

Vertical Path Profile Display

Aircraft Track

Current ownship position

Steam gauge overlay replicates pilot instruments

Selectable data field used to color tracks

Adjustable legend for track colors

Figure 8. Query Page of CATS Used to Include or Exclude Data Records Based on User Defined Query

4.B.2-10

Waveform Window

Removing Sections of

Data

While the detection of bad channels feature automatically analyzes and picks out bad data from individual channels, sometimes data can be suspect for a period of time for all channels due to interference from an external source. CATS allows for the manual selection and removal of time sections of data by the user if necessary.

1. Left click on waveform window and drag to highlight a section of the database

2. Right click on highlighted region to pull up options

3. Press ‘Exclude Region’

4. Repeat 1-3 for as many sections in the database as necessary then right click and press ‘Save to Meta’

5. Press ‘Enable Query

**If a meta file has already been saved right click on Waveforms, press ‘Load from Meta’, then press ‘Enable Query’

Figure 9. Waveform Viewing Page

Workload Scoring with CATS CATS was used to determine the cognitive

workload that was expended by the participants during a number of standardized cognitive tasks. One of the tasks consisted of three levels of workload, low, medium, and high. Participants viewed integer numbers from 0 to 9 appearing in 1 second intervals on a computer screen. For the low workload task, the participants had to push a button when the number 5 appeared, for the medium task, they had to push the button when three even numbers appeared in a row, and for the hard task, they had to push the button each time a number appeared that appeared three presentations ago. The results for 25 participants are shown in Figure 12 and indicate excellent diagnostic power of cognitive task workload (as specified by task difficulty) using EEG power in the alpha and beta bands.

We also often give users a digit recall task to simulate different levels of cognitive workload for benchmarking purposes. This task measures the cognitive effort that is required to memorize

numbers of increasing lengths for a duration of 30 seconds. We usually administer this digit recall task for numbers of 5 to 9 digits in length.

Figure 13 shows the ratio of EEG Beta/Alpha power in the occipital regions of one subject performing the digit recall task. It appears that this participant expended considerably more effort to remember numbers that were 8 and 9 digits long when compared to shorter sequences. The reader should note that we use bar graphs of EEG power in clinical bands such as those shown in this paper solely for documentation purposes. CATS does not perform its operator state classification on the basis of a small number of frequency spectra such as those delineated in clinical bands. CATS calculates EEG power at all locations on the scalp in all available frequency bands simultaneously and in real-time. This data is then exposed onto the neural network classifier for training of the network or for real-time classification of operator state.

4.B.2-11

Cognitive Task Workload Difficulty

Occ

ipita

l Alp

ha a

mpl

itude

HighMediumLow

225

200

175

150

125

100

75

50

N=25 Participants

a. Alpha Power in Occipital Region for 25 Participants

Cognitive Task Workload Difficulty

Occ

ipit

al B

eta

Am

plit

ude

HighMediumLow

200

150

100

50

0

N=25 Participants

a. Beta Power in Occipital Region for 25 Participants

Note: N=25 participants. Cognitive Loading Task, low, medium, and high workload. Participants viewed integer numbers from 0 to 9 appearing in 1 second intervals on a computer screen. For the low workload task, the participants had to push a button when the number 5 appeared, for the medium task, they had to push the button when three even numbers appeared in a row, and for the hard task, they had to push the button each time a number appeared that appeared three presentations ago.

Figure 10. EEG Power in Occipital Region as a Function of Cognitive Loading

4.B.2-12

0.6

0.62

0.64

0.66

0.68

0.7

0.72

0.74

0.76

0.78

0.8

DR-5 DR-6 DR-7 DR-8 DR-9

Digit Recal, Number of Digits

Rat

io o

f Bet

a/A

lpha

Note: DR means Digit Recall and the number denotes the number of digits held in short term memory for 30 seconds

Figure 11. Ratio of Occipital Beta over Alpha Power in One Participant Performing a Digit Recall Task

Discussion We embarked on our multi-sensory

physiological research in a real-flight environment, knowing that we needed a very robustly developed data collection apparatus. Sorties on flight test platforms are very expensive and the research equipment must be integrated such that it can reliably operate in the demanding environment of real flight. Moreover, the equipment must be designed and implemented according to stringent airworthiness certification rules. Most clinical data collection devices need to be specifically flight-hardened for use in the flight test environment and the installation should comply with standards and best practices applicable for avionics equipment.

The Cognitive Avionics Tool Set (CATS) is a hardware and software architecture that grew out of this need and was developed to provide a reliable platform for operator state characterization.

Synchronized data collection is essential and it is best to perform the synchronization through a local area network in real-time during data collection. Multi-sensory data should be synchronized properly in real-time during data collection to avoid a huge burden on the researchers during data post-processing. Also, data that is not accurately synchronized will introduce inaccuracies between the occurrence of performance events and

the neurophysiological measures. This is, of course, not acceptable for any real-time performance monitoring systems. The CATS architecture was specifically designed to synchronize multi-sensory data and provide the ability to tag events exactly when they happen. CATS is also able to incorporate video and audio data in the same unified data structure. This provides the researchers with the ability to record voice narrations that can be useful during after action review.

Flight testing of Cognitive Avionics systems is challenging and can be very rewarding as the technology application is tested in an actual target environment. Avionics is the major growth factor in new airframes.

References [1] Hellenic republic Ministry of transport & communications, Air accident investigation & aviation safety board, Helios airways flight CY522, Boeing 737-31s, At Grammatiko, Hellas on 14 August 2005

[2] Schnell T., Keller, M., & Macuda, T. (2007a, October). Application of the Cognitive Avionics Tool Set (CATS) in airborne operator state classification. Paper presented at the Augmented Cognition International Conference, Baltimore, MD.

4.B.2-13

[3] Schnell, T., Keller, M., & Macuda, T. (2007b, April). Pilot state classification and mitigation in a fixed and rotary wing platform. Paper presented at the Aerospace Medical Association (ASMA) annual conference, New Orleans, LA.

[4] Schnell, T., Macuda, T., & Keller, M. 2008, Operator state classification with the cognitive avionics tool set. In D. Schmorrow & K. Stanney (Eds.), Augmented Cognition: A practitioner’s guide.

[5] Schnell, T., Macuda, T., & Poolman, P. (2006, October). Toward the cognitive cockpit: Flight test platforms and methods for monitoring pilot mental state. Paper presented at the Augmented Cognition International Conference, San Francisco, CA

[6] Schnell T., Keller M., Cornwall R., Walwanis-Nelson, M. Tools for Virtual Environment Fidelity Design Guidance: Quality of Training Effectiveness Assessment (QTEA) Tool, Phase I STTR Report N00014-07-M-0345-0001AC, Contract Number N00014-07-M-0345; 2007a.

Acknowledgements The Operator State Characterization Research

described in this paper are funded by the NASA Langley Research Center (LaRC) Aviation Safety Program under contract NNL07AA00A. The Cognitive Pilot Helmet (CPH) prototype was funded by Rockwell Collins.

4.B.2-14