Upload
buikiet
View
221
Download
0
Embed Size (px)
Citation preview
Training Effectiveness of Flight Simulatorswith Outside Visual Cues
Miguel Freitas da Silva Mendes
Thesis to obtain the Master of Science Degree in
Aerospace Engineering
Supervisor: Prof. Agostinho Rui Alves da Fonseca
Examination Committee
Chairperson: Prof. João Manuel Lage de Miranda Lemos
Supervisor: Prof. Agostinho Rui Alves da Fonseca
Member of the Committee: Prof. José Raúl Carreira Azinheira
October 28, 2016
Acknowledgments
Conducting a research project like the one behind this dissertation was something I looked
forward to ever since I started my academic path, back in 2010. When choosing my Grad-
uation Project to obtain the Master degree on the Aerospace Engineering Faculty at Delft
University of Technology, this topic was proposed by my thesis advisor dr.ir. Daan Pool and
to him I must dedicate the first acknowledgment of this work. The door of dr.ir. Pool’s office
was always opened whenever I ran into a trouble during the last 7 months and his consistent
guidance and immense knowledge and experience were fundamental to elaborate this work.
I must also thank to senior researcher engineer Peter Zaal, of the NASA Ames Research
Center at San Jose State University, for his availability to steer me in the right direction with
his fruitful and insightful inputs given in several moments of the work developed.
I would further like to express a note of gratefulness to the 22 people who volunteered to
perform this experiment. Their commitment to this project, with each of them spending a
total of 8 hours on the simulator, was remarkable.
On a personal level, a significant thank-you must be expressed to Joao Paulo, for his valuable
friendship and for being my travel partner in the journey this Master’s program we both
followed was. I must also dedicate the deepest appreciation note to Mariana for her crucial
presence and importance in my life. Last but definitely not least I must express my very
profound gratitude to my parents and my sister, my aunt and my grandparents, for their
continuous encouragement throughout my years of study, unfailing support, and ultimately
for providing me with this opportunity of completing my studies far from their so loved home.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
ii Acknowledgments
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
Abstract
To control an aircraft, human pilots rely on their visual and somatosensory systems to perceive
their self-motion through the world and thereby control their aircraft. Nowadays, pilot acquire
such low-level control skills almost exclusively during simulator-based training. To study
this learning process and investigate the effectiveness of simulator-based training procedures,
transfer-of-training experiments are performed to evaluate from a cybernetic perspective how
the pilots control behavior changes during learning. Some simulator manufacturers claim
that motion cues presented on the outside visual may substitute for physical motion cues,
however the effects of outside visual cues, as typically available in flight simulators, on control
skill development remains largely unknown. To investigate this, a quasi-transfer-of-training
experiment with twenty fully task-naive participants was conducted in the SIMONA Research
Simulator at Delft University of Technology. The main hypothesis for the experiment was that
training with outside visual cues would ease the development of a multimodal control strategy
and the effective use of physical simulator motion feedback for control. It was found that,
while outside visual cues do improve task performance, they result in a control strategy that
shows no meaningful transfer to a motion-base condition, suggesting that physical motion
cues are still very important to supply during the initial simulator-based training of basic
control skills.
Keywords: Flight Simulator, Cybernetics, Outside Visual Cues, Manual Control, Transfer
of Training.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
Resumo
Um piloto humano utiliza a sua visao e o seu sistema somatossensorial para percepcionar o
movimento da aeronave, usando essa informacao para a controlar. Hoje em dia, os pilotos
adquirem aptidoes de pilotagem com recurso tendencialmente exclusivo a treino em simu-
ladores de voo. Para entender de que forma a estrategia de controlo dos pilotos muda durante
este processo de treino em simuladores de voo, bem como para avaliar a sua eficiencia numa
perspectiva cibernetica, realizam-se experiencias de transferncia de treino. Alguns fabricantes
de simuladores alegam que estımulos de movimento apresentados no cenario exterior podem
substituir a presenca de movimento fısico do simulador. Contudo, os efeitos da existencia
de um cenario visual exterior durante o treino de pilotos em simuladores permanecem em
grande parte desconhecidos. Para investigar este aspecto, uma experiencia de treino com 20
participantes sem experiencia previa de pilotagem foi conduzida no simulador de investigacao
SIMONA, para testar a hipotese de que a presenca de um cenario visual exterior durante
o treino dos sujeitos inicialmente inexperientes facilitaria a habituacao a uma situacao onde
existisse movimento fısico do simulador, ja que o cenario visual permitiria a utilizac de feed-
back para controlo. Foi concluido que, por um lado, o cenario exterior melhora o desempenho
dos pilotos no simulador, mas a sua presenca induz uma estrategia de controlo que nao se
transfere positivamente para uma condicao com movimento fısico do simulador. Isto sugere
que a presenca de movimento fısico do simulador de voo e o estımulo mais importante a
providenciar nas etapas iniciais de treino de pilotos em simuladores de voo.
Palavras-chave: Simulador de Voo, Cibernetica, Cenario Visual Exterior, Controlo Manual,
Transferencia de Treino.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
Acronyms
DCF Disturbance Crossover Frequency
DPM Disturbance Phase Margin
DUT Delft University of Technology
FC Fourier Coefficients
GN Gauss-Newton
MISC Misery Scale
MLE Maximum Likelihood Estimation
NV No Visuals
PFD Primary Flight Display
RMS Root Mean Square
SISO Single Input Single Output
SRS SIMONA Research Simulator
TCF Target Crossover Frequency
TPM Target Phase Margin
V Visuals
VAF Variance Accounted For
VAR Variance
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
List of Symbols
Greek Symbols
∆ Fourier transform of system input
δ system input
ǫ estimation error
Γ coherence
ωc crossover frequency
ω radial frequency
φm phase margin
φ roll angle
ρ Pearson’s correlation coefficient
σ standard deviation
τ time delay
Θ parameter vector
ζ damping
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
x List of Symbols
Roman Symbols
A amplitude/system matrix
B input matrix
C output matrix
D feedthrough matrix
E Fourier transform of the error
e error
F Fourier transform of the forcing function/learning curve rate
f forcing function
H frequency response function
j imaginary unit
K gain
k loop counter
m measurement sample number
N Fourier transform of the remnant
n remnant
S power spectral density function
T time constant
t time
U Fourier transform of the control input
u control input
X Fourier transform of the output
x output
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
Subscripts
φ roll response
c controlled element
d disturbance
e error response
lag lag
lead lead
nm neuromuscular system
lc learning curve model
ol open-loop
scc semi-circular canals
t target
u control output
v visual
x system output
xii List of Symbols
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
Contents
Acknowledgments i
Abstract iii
Resumo v
Acronyms vii
List of Symbols ix
1 Introduction 1
1-1 Dissertation Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Experiment Preparation 5
2-1 Identification Methods in Offline Simulations . . . . . . . . . . . . . . . . . . . . 5
2-1-1 Experiment Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2-1-2 Identification Using Fourier Coefficients . . . . . . . . . . . . . . . . . . 7
2-1-3 Identification using Maximum Likelihood Estimation . . . . . . . . . . . 9
2-1-4 Offline Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2-2 Pilot Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2-2-1 Controlled Element Dynamics . . . . . . . . . . . . . . . . . . . . . . . . 15
2-2-2 Outside Visual Cues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3 Experimental Methods 31
3-1 Control Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3-2 Forcing Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3-3 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3-4 Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3-5 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
xiv Contents
3-6 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3-6-1 Human Operator Modeling . . . . . . . . . . . . . . . . . . . . . . . . . 39
3-6-2 Other Dependent Variables . . . . . . . . . . . . . . . . . . . . . . . . . 41
3-6-3 Learning Curve Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3-6-4 Statistical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3-7 Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4 Experimental Results 45
4-1 Tracking Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4-2 Control Activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4-3 Human Operator Modeling Results . . . . . . . . . . . . . . . . . . . . . . . . . 50
4-3-1 Model Fits and Describing Functions . . . . . . . . . . . . . . . . . . . . 50
4-3-2 Coherence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4-3-3 Variance Accounted For . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4-3-4 Human Operator Model Parameters . . . . . . . . . . . . . . . . . . . . 54
4-3-5 Crossover Frequencies and Phase Margins . . . . . . . . . . . . . . . . . 57
5 Discussion 59
6 Conclusions 63
6-1 Recommendations for Future Training Experiments . . . . . . . . . . . . . . . . 64
A Processing Experimental Results 71
A-1 Excluding Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
A-1-1 Excluded Subject 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
A-1-2 Excluded Subject 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
A-2 Solving Identification Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
A-3 Variations on Identification Methods . . . . . . . . . . . . . . . . . . . . . . . . 82
A-3-1 Identification on Multiple Consecutive Runs . . . . . . . . . . . . . . . . 82
A-3-2 Single-Channel for Training of Group V . . . . . . . . . . . . . . . . . . 84
A-3-3 Double-Channel for Training of Group NV . . . . . . . . . . . . . . . . . 86
B Experiment Documents 89
B-1 Call for Volunteers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
B-2 Consent Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
B-3 Experiment Briefing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
B-4 Experiment Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
B-5 Experiment Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
List of Figures
2-1 Schematic representation of the pitch tracking task. . . . . . . . . . . . . . . . . 6
2-2 Simulated frequency response of human operator in run 175. . . . . . . . . . . . 12
2-3 Frequency response of target and disturbance open loops for run 175. . . . . . . 13
2-4 Schematic representation of the roll tracking task. . . . . . . . . . . . . . . . . . 15
2-5 Compensatory tracking display shown in the PFD. . . . . . . . . . . . . . . . . . 15
2-6 Frequency response of the three considered dynamics. . . . . . . . . . . . . . . . 17
2-7 RMS value of tracking error and control activity in the three dynamics tested. . . 19
2-8 Subject 1 frequency response for the three dynamics tested. . . . . . . . . . . . 20
2-9 Subject 1 target open loop frequency response for the three dynamics tested. . . 20
2-10 Target open loop crossover frequency and phase margin of the tested dynamics. . 20
2-11 Schematic representation of the roll tracking task. . . . . . . . . . . . . . . . . . 23
2-12 Checkerboard visual scene. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2-13 Realistic visual scene. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2-14 Flow-field visual scene. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2-15 RMS value of tracking error and control in the visual testing. . . . . . . . . . . . 26
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
xvi List of Figures
2-16 Frequency response of Subject 1 for C2 (Visuals On, Motion Off). . . . . . . . . 27
2-17 Open loop frequency response of Subject 1 for C2 (Visuals On, Motion Off). . . 28
2-18 Crossover frequencies and phase margins in the visual testing. . . . . . . . . . . 29
3-1 Schematic representation of the roll tracking task. . . . . . . . . . . . . . . . . . 32
3-2 Simulator cockpit, central display and the out-of-the-window scene. . . . . . . . 32
3-3 Frequency response of the controlled element dynamics. . . . . . . . . . . . . . . 33
3-4 Quasi-transfer-of-training experiment design. . . . . . . . . . . . . . . . . . . . . 36
3-5 The SIMONA Research Simulator at TU Delft. . . . . . . . . . . . . . . . . . . 37
4-1 Tracking error variance and the disturbance, target, and remnant components. . 46
4-2 Average control input variance. . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4-3 Frequency Response of Error and Roll channels in the human operator control model. 51
4-4 Average coherence for the initial and final runs in training and evaluation phases. 53
4-5 Average Variance Accounted For of the estimated model. . . . . . . . . . . . . . 53
4-6 Average estimated parameters defining the error response. . . . . . . . . . . . . 54
4-7 Average estimated parameters defining the human operator neuromuscular system. 55
4-8 Average estimated parameters defining the roll response. . . . . . . . . . . . . . 55
4-9 Average disturbance and target crossover frequencies and phase margins. . . . . 57
A-1 Tracking error and control input variances for Excluded Subject 1. . . . . . . . . 72
A-2 VAF of the identified model for Excluded Subject 1. . . . . . . . . . . . . . . . . 73
A-3 Estimated parameters of the model for Excluded Subject 1. . . . . . . . . . . . . 74
A-4 Tracking error and control input variances for Excluded Subject 2. . . . . . . . . 75
A-5 Detail of the tracking error variance for Excluded Subject 2. . . . . . . . . . . . 75
A-6 VAF of the identified model for Excluded Subject 2. . . . . . . . . . . . . . . . . 76
A-7 Estimated parameters of the model for Excluded Subject 2. . . . . . . . . . . . . 77
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
List of Figures xvii
A-8 Bode plots comparing different model estimates. . . . . . . . . . . . . . . . . . . 79
A-9 Overview of the experiment runs identifying the origin of the final identified model. 81
A-10 VAF of the 5 runs model of Group V. . . . . . . . . . . . . . . . . . . . . . . . 83
A-11 5-runs estimation of error response model parameters of Group V. . . . . . . . . 83
A-12 5-runs estimation of neuromuscular system parameters of Group V. . . . . . . . . 83
A-13 5-run estimation of roll response model parameters of Group V. . . . . . . . . . 84
A-14 VAF of the single-channel model for training of Group V. . . . . . . . . . . . . . 85
A-15 Error response model parameters of Group V with different model structures. . . 85
A-16 Neuromuscular system parameters of Group V with different model structures. . . 86
A-17 VAF of the double-channel model for training of Group NV. . . . . . . . . . . . 87
A-18 Error response model parameters of Group NV with different model structures. . 88
A-19 Neuromuscular system parameters of Group NV with different model structures. . 88
A-20 Roll response model parameters of Group NV with a double-channel structures. . 88
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
xviii List of Figures
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
List of Tables
2-1 Parameters of the human operator control model for run 175. . . . . . . . . . . 12
2-2 Parameter estimates for both subjects for three dynamics tested. . . . . . . . . . 19
2-3 Experimental conditions for the outside visual cues pilot experiment. . . . . . . . 22
3-1 Experiment forcing function (disturbance and target) data . . . . . . . . . . . . 35
4-1 Learning curve parameters and statistical analysis for total tracking error. . . . . 46
4-2 Learning curve parameters and statistical analysis for disturbance tracking error. . 46
4-3 Learning curve parameters and statistical analysis for target tracking error. . . . . 46
4-4 Learning curve parameters and statistical analysis for remnant tracking error. . . 47
4-5 Learning curve parameters and statistical analysis for control input. . . . . . . . 49
4-6 Parameters for the evaluation phase learning curves shown in Figs. 4-6 and 4-8. . 55
A-1 Comparison between obtained models. . . . . . . . . . . . . . . . . . . . . . . . 80
A-2 Overview of the origin of the final considered model. . . . . . . . . . . . . . . . 80
B-1 Experiment scheduling - testing week. . . . . . . . . . . . . . . . . . . . . . . . 94
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
xx List of Tables
B-2 Experiment scheduling - first week. . . . . . . . . . . . . . . . . . . . . . . . . . 95
B-3 Experiment scheduling - second week. . . . . . . . . . . . . . . . . . . . . . . . 96
B-4 Experiment scheduling - third week. . . . . . . . . . . . . . . . . . . . . . . . . 97
B-5 Experiment scheduling - forth week. . . . . . . . . . . . . . . . . . . . . . . . . 98
B-6 Experiment scheduling - fifth week. . . . . . . . . . . . . . . . . . . . . . . . . . 99
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
Chapter 1
Introduction
In order to control and steer any vehicle, whether it is a bicycle, a car or an aircraft, humans
depend on their sensory systems to perceive the surrounding reality and thereby gather infor-
mation relevant for control. Visual and motion stimuli are the most relevant control inputs
and control proficiency is attained by training responses to what the visual and vestibu-
lar systems perceive [1]. In case of an aircraft, pilots’ learning process usually includes a
simulator-based phase in which simulators replicate flight reality for pilots to develop their
control skills. The control skills acquired by a pilot in his simulator training are brought into
use when transferred to a real-world setting as flying an actual aircraft. Understanding what
changes in pilots’ responses to their sensory systems inputs throughout their learning process,
together with why and how those changes occur, will result in improved flight simulators, pilot
training, and ultimately pilot skills.
A cybernetic approach to understand these changes consists on transfer-of-training experi-
ments, in which the transfer of control behavior acquired in a training condition (e.g., a flight
simulator) to the evaluation setting (e.g., a real aircraft) is investigated and directly assessed.
However, given the impracticability in performing transfer to real settings, the majority of
the studies performed are in fact quasi-transfer-of-training experiments, where the evaluation
setting is not true reality but a more realistic simulation environment [2]. The learning of skill-
based manual control is characterized by the development of low-level automated responses
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
2 Introduction
to continuous environmental feedback signals [3], and the extent to which trained behavior
transfers to a different environment is mainly defined by the environmental dependency of the
applied skills [4, 5]. Multiple transfer-of-training experiments were performed to understand
what are the effects of different types of simulator cues on humans’ learning of control behav-
ior and how these cues affect skill transfer. Most of them focused on the training effectiveness
of motion cues, having found that motion feedback is required for effective simulator-based
training of manual control skills [6, 7, 8]. This happens because motion feedback strongly
influences human operators’ behavior, especially when the controlled dynamics require lead
equalization [9, 10, 11].
Recent studies with compensatory tracking tasks have shown that outside visual cues are
utilized by human operators to support a human feedback control organization similar to
the one observed in tasks with physical motion cues [12]. It was proven that the presence
of a strong outside visual scene provides lead information on the controlled dynamics in a
similar way as achieved by the physical motion feedback, though not as effectively [1, 13,
14]. If these findings are taken into consideration from a perspective of simulator-based
training, similarities in the way human operators deal with both sensory inputs suggest that
outside visual cues might be used for initial simulator-based training, as they might create
and establish a feedback channel in the human operator without the need of actual physical
motion cues. At this point, such transfer has never been studied explicitly and it is thus
hypothesized that the feedback channel created by the existence of an outside visual scene is
effective in easing the developing of manual control skills in a motion condition.
The goal of the research conducted in this project is to discover to which extent visual cues
are effective in developing multimodal control skills during simulator-based pilot training.
To achieve this goal, a quasi-transfer-of-training experiment was conducted in the SIMONA
Research Simulator at Delft University of Technology, in Delft, the Netherlands, and it is
hereby described and analyzed. Twenty fully task-naive participants performed the exper-
iment, which consisted on a compensatory roll attitude tracking task, similar to multiple
earlier training and tracking experiments [8, 15, 16, 13]. Subjects were divided in two ex-
perimental groups and performed 100 training runs, either with the simulator outside visual
system off (“no-visuals”) or on (“visuals”). After this training phase, participants were trans-
ferred to the evaluation setting, where both groups performed 100 more runs with pure roll
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
1-1 Dissertation Outline 3
motion provided by the simulator and without an outside visual scene. Each run performed
was subsequently analyzed in terms of tracking performance, control activity, human oper-
ator control behavior, among other derived data. The overall evolution of the quantities of
interest throughout the runs, together with the comparison of group averaged results, pro-
vided insight into the efectiveness of outside visual cues in using physical motion cues during
simulator-based training of manual piloting skills.
1-1 Dissertation Outline
This dissertation is structured as follows. The main part of this dissertation starts in Chapter
2 in which the most important aspects of the experiment preparation are given, with an
emphasis on the offline simulations made to prepare the algorithms used and in the pilot
experiments to define the experiment design. Afterwards, in Chapter 3, the methods, the
organization of the experiment, and the hypotheses are described. Chapter 4 contains the
results of the experiment grouped in three different categories. Tracking performance is
presented in section 4-1 and control activity in section 4-2. The human operator modeling
results are shown in section 4-3. A discussion follows in Chapter 5 and the main part of
the dissertation ends with conclusions in Chapter 6, together with some recommendations
regarding possible future training experiments.
In the final part of this dissertation, some appendices are included, covering further aspects
of the work developed. In appendix A a description on the approach followed to deal with
problems in the processing of the results is given. This dissertation ends with appendix B, in
which some important experiment documents are presented.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
Chapter 2
Experiment Preparation
To perform a cybernetic experiment as the quasi-transfer-of-training study conducted in this
work, an extensive and careful preparation should be made with an emphasis on firstly the
available background research on the topic and secondly on the design of the experiment itself.
In this Chapter, the preparation of the final experiment is given, with the steps followed to
justify the final experiment design and the tools used to process the data of the experiment.
2-1 Identification Methods in Offline Simulations
The first steps of the experiment preparation consisted on extensively studying the previous
training experiment performed in the Control & Simulation group on the Aerospace Engi-
neering Faculty at Delft University of Technology, a quasi-transfer-of-training experiment
developed by Pool et al. [8] in the SIMONA Research Simulator (SRS). The main conclusion
reached in this study was that motion feedback is required for effective initial simulator-based
training of skill-based manual control. The methods followed in the referred research, mainly
the experiment design and the data processing, were carefully analyzed to prepare the cur-
rent experiment. The comprehension and development of the identification techniques to be
applied in the current experiment were made using a simulation of the referred experiment,
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
6 Experiment Preparation
as a way of validating the algorithms developed. An overview of the experiment is given in
Sec. 2-1-1 to contextualize the reader for the following sections, in which two identification
methods are detailed, in Sec. 2-1-2 and Sec. 2-1-3, and in Sec. 2-1-4 the results of the
application of these methods in offline simulations are shown.
2-1-1 Experiment Overview
The experiment conducted by Pool et al. [8] in the SRS was meant to quantify the effects
of simulator motion feedback on the training of skill-based human operator control behavior.
A quasi-transfer-of-training experiment was performed using 24 task-naive participants who
were divided over two groups and were trained in performing a skill-based compensatory pitch
tracking task. The first group was trained in a fixed-base setting and transferred to a moving-
base condition; the second group was trained with motion feedback and then transferred to
the fixed-base condition. The task performed consisted on minimizing the pitch tracking
error which was shown in a visual display as a deviation from the current pitch angle θ
and the tracking signal ft. If motion is available, direct feedback of θ is perceived by the
human operator through motion cues. A disturbance signal fd directly affects the controlled
dynamics, as it is summed to the human operator input u. The initially task-naive participants
controlled the elevator-to-pitch dynamics of a Cessna Citation I, given in Eq. (2-1). A
representation of the pitch tracking task can be found in Fig. 2-1.
Hθ,δe(s) = 10.62s+ 0.99
s(s2 + 2.58s+ 7.61)(2-1)
Figure 2-1: Schematic representation of the tracking task performed by Pool et al. [8].
To model the human operator response to visual and motion feedback, a quasi-linear model
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
2-1 Identification Methods in Offline Simulations 7
is used, in which the behavior of the human operator is divided in a linear part and the
remaining non-linear behavior is captured by a noise signal, called the remnant. To model
the human operator response to motion feedback independently of the visual feedback, a
multi-channel model structure is used, as shown in Fig. 2-1. When motion is not available, a
single channel model is used (Hpm(s)=0). The structure of the human operator response in
each of the two channels is shown in Eq. (2-2) and Eq. (2-3).
Hpv(s) = Kv(Tleads+ 1)2
(Tlags+ 1)e−sτvHnm(s) (2-2)
Hpm(s) = s2Hscc(s)Kme−sτmHnm(s) (2-3)
Where Hnm(s) represent the neuromuscular system dynamics and Hscc(s) represent the semi-
circular canals system dynamics, given respectively in Eqs. (2-4) and (2-5).
Hnm(s) =w2nm
s2 + 2ζnmωnms+ w2nm
(2-4)
Hscc(s) =0.11s+ 1
5.9s+ 1(2-5)
2-1-2 Identification Using Fourier Coefficients
The first method used to obtain a description of the human operator control behavior is a
frequency-domain approach using the Fourier coefficients of the discrete Fourier transform
of the time signals recorded in the SRS (or in this Chapter, the time signals obtained in
a simulation of the task). This black-box method allows the estimation of the frequency
response of a controller in a tracking task [17, 18].
Single-Input-Single-Output
The easiest systems to identify are single-input-single-output (SISO) systems, which be-
comes the human operator case if the motion feedback channel in Fig. 2-1 is not considered
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
8 Experiment Preparation
(Hpm(s) = 0), holding the following relation in the frequency domain for the human operator
control behavior:
U(s) = Hpv(s)E(s) +N(s) (2-6)
If this relation is considered in the frequencies of the target signals, then the signal to noise
ratio becomes high due to the high power of those signals in the considered frequencies. For
that reason, the remnant noise can be ignored in those frequencies, yielding the following
relation for the human describing function, at the discrete frequencies composing the target
signal:
Hpv(jωt) =U(jωt)
E(jωt)(2-7)
Two-Inputs-Single-Output
For a multi-channel model structure, the output signal is related to the input signals as follows,
considering already the Fourier transform of the considered signals at the target signals:
U(jωt) = Hpv(jωt)E(jωt)−Hpm(jωt)θ(jωt) (2-8)
In this equation, there are two unknowns, Hpv(jωt) and Hpm(jωt), which means a second
equation should be added to the picture to allow solving both unknowns. This is the reason
why two forcing functions are needed in the experiment task, target and disturbance, because
interpolating the signals at the disturbance frequencies to the target frequencies holds a second
set of signals valid at the target frequencies. The estimates of the describing functions for a
two-channel control task are given by Eqs. (2-9) and (2-10) as shown by Nieuwenhuizen et
al. [19]:
Hpv(jωt) =U(jωt)θ(jωt)− U(jωt)θ(jωt)
E(jωt)θ(jωt)− E(jωt)θ(jωt)(2-9)
Hpm(jωt) =U(jωt)E(jωt)− U(jωt)E(jωt)
E(jωt)θ(jωt)− E(jωt)θ(jωt)(2-10)
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
2-1 Identification Methods in Offline Simulations 9
The tilde in this equation denotes interpolation from the target to the disturbance input fre-
quencies. These equations are only valid at the input frequencies of the target forcing function,
but exchanging the frequency subscripts from t to d holds the expression for identification at
the disturbance input frequencies.
One final detail, for the interpolation of the complex Fourier coefficients to make sense the
initial phase of the input signals has to be removed, using Eq. (2-11), where X denotes any
signal (can be U , E, θ), and Ft(ωt) is the Fourier coefficient of the target forcing function at
the specific target input frequencies. The same relation is valid for the disturbance signal.
X∗(ωt) = X(ωt)e−j∠Ft(ωt) (2-11)
Regarding this method, it must be seen that it provides limited knowledge about the system
because the identified system is only known at very specific frequencies. The system remains
unknown in other frequencies, and if in those frequencies the signal-to-noise ratio is low,
remnant noise plays an important role in the human model and therefore these estimates
fail to provide the whole picture on the human control behavior. To reduce noise and non-
linearities contribution, this method yields better results if applied to multiple averaged runs.
2-1-3 Identification using Maximum Likelihood Estimation
The previous method holds a general description of the human operator frequency response
but it lacks information about the model structure itself. It is desirable to estimate the fre-
quency response using a parametric model, which yields insight in the physical characteristics
of human response behavior. To do so, a Maximum Likelihood Estimation (MLE) procedure
is applied, which is extensively described in P. M. T. Zaal et al. [20]. This method offers sat-
isfiable statistical properties, namely the convergence of the estimates to the true parameter
set and the variance of the estimates reducing to the lower Cramer-Rao bound as the sample
size increases.
Being a time-domain parameter estimation method, this algorithm requires a state-space rep-
resentation of the human operator dynamics. Considering a double-channel control loop with
feedback in the error and in the controlled angle, and denoting with Θ the set of parameters
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
10 Experiment Preparation
defining the model, the state-space description is, following Ref. [20]:
˙x =
Ae(Θ) 0
0 Aθ(Θ)
x+
Be(Θ) 0
0 Bθ(Θ)
e(t)
θ(t)
(2-12)
u(t) =[
Ce(Θ) −Cθ(Θ)]
x(t) +[
De(Θ) −Dθ(Θ)]
e(t)
θ(t)
+ n(t) (2-13)
The Ae,θ, Be,θ, Ce,θ and De,θ matrices are obtained by converting the respective transfer
functions to the controller canonical form and replacing the parameters that define each
channel by the current estimate. The estimation error for a certain sample number k can be
calculated with:
ǫ(k|θ) = uexp(k|Θ)− u(k|Θ) (2-14)
Where uexp is the recorded signal and u is the modeled control signal, i.e., the output signal
calculated by Eq. (2-13).
The maximum likelihood estimate of the parameters is finally obtained by optimizing the
criterion:
ΘML = argminΘ
[
m
2lnσ2
n +1
2σ2n
m∑
k=1
ǫ2(k)
]
(2-15)
This becomes a highly non-linear optimization problem, due to the many degrees of freedom
of the model and the complementary nature of both feedback paths. To solve the optimization
of the criterion given in Eq. (2-15), a two-step process is taken. Firstly, a genetic algorithm
is applied to find a parameter set that is close to the global optimum, and subsequently this
set serves as an initial estimate of a Gauss-Newton algorithm, a gradient-based optimization
method. A detailed explanation of this method is given in [20].
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
2-1 Identification Methods in Offline Simulations 11
2-1-4 Offline Simulations
The results of the experiment performed by Pool et al. [8] were utilized in offline simulations
to develop and validate the identification techniques presented in the previous Section. These
offline simulations were made in MatLab’s simulation environment, Simulink. The experiment
diagram shown in Fig. 2-1 was reproduced in Simulink, with the human operator model being
characterized by the parameters found by Pool et al. [8]. The simulated time signals of the
tracking error e, the pitch angle θ and the control input u were subsequently processed in the
identification techniques described to determine the parameters that were initially defined in
Simulink, providing thus a validation on the algorithms implemented.
Training and evaluation phases of one of the groups of the referred experiment were considered
and replicated in the offline simulations, meaning both single channel and double channel
identification was performed. The describing learning curves obtained in the experiment were
utilized to provide the parameters of the human operator response. Noise was not introduced
in the simulation, meaning the remnant noise which is part of the human operator control
behavior was not included in the offline simulations performed. This means the human
operator is simulated as a fully linear system, which will lead to an expected almost perfect
match between both the Fourier coefficients method and the Maximum Likelihood Estimation.
In Figure 2-2 the results of the double-channel identification for the last run in the evaluation
phase are shown. The blue line represents the identified model and the red markers the
Fourier coefficients of the human operator describing functions. An almost perfect fit is
visible, validating the estimation of human behavior provided by both methods. One remark
should be done considering the motion response. The considered structure for the motion
response was the one shown in Eq. (2-3), without the double pure derivative term s2. This
means the input to the system was not the pitch angle, but the pitch acceleration, i.e., a
feedback of an internal state of the controlled dynamics was made to the human operator.
This is done to avoid having a non-proper transfer function in the motion channel response,
as a non-proper transfer function forces MatLab to perform derivation on the time signals
during the identification algorithm, with numerical issues arising. Therefore, in Figs. 2-2(c)
and 2-2(d), the frequency response corresponds to the transfer function shown in Eq. (2-3),
without s2.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
12 Experiment Preparation
Fourier Coefficients
MLE Model
replacemen
|Hpe(jω)|,—
10-1 100 101 10210-2
10-1
100
101
102
(a) Error response - magnitude.
∠(H
pe(jω)),deg
10-1 100 101 102-630
-540
-450
-360
-270
-180
-90
0
90
180
(b) Error response - phase.
|Hpm(jω)|,—
10-1 100 101 10210-2
10-1
100
101
102
(c) Motion response - magnitude.
∠(H
pm(jω)),deg
10-1 100 101 102-630
-540
-450
-360
-270
-180
-90
0
90
180
(d) Motion response - phase.
Figure 2-2: Frequency response of human operator for run 175 (evaluation phase), using simu-lated time signals.
Furthermore, in Table 2-1, the parameters provided by the learning curve (which were used in
Simulink to obtain the time signals on which identification was performed) and the parameter
estimates that were the result of MLE identification technique are shown. Giving that both
lines are highly similar, it can be concluded that the algorithm is properly estimating the
parameters that define human operator control behavior.
Table 2-1: Parameters of the human operator control model for run 175, in the learning curvefound by Pool et al. [8] and in the MLE technique using a offline simulation of the experimentperformed.
Ke, — Tlead, s Tlag, s τe, s ωnm, rad/s ζnm, — Km, — τm, s
Learning Curve 4,3608 0,2923 0,7213 0,2334 12,5000 0,3000 3,1003 0,1812MLE 4,3608 0,2923 0,7213 0,2333 12,4984 0,3001 3,1004 0,1812
In Figure 2-3 the frequency response of target and disturbance open loops is shown, calculated
using the Fourier coefficients of the signals that define respectively the target and disturbance
open loops, as given in Eqs. (2-16) and (2-17) (where ∆e represents the Fourier Transform of
the time signal corresponding to the elevator command, δe), and recurring to the parameters
determined by the MLE identification algorithm, according to Eqs. (2-18) and (2-19). Again,
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
2-1 Identification Methods in Offline Simulations 13
an almost perfect match between both methods is obtained in the four Bode plots, validating
the procedures followed.
Holt(jωt) =X(jωt)
E(jωt)(2-16)
Hold(jωd) = −U(jωd)
∆e(jωd)(2-17)
Holt(s) =Hpe(s)Hc(s)
1 +Hpm(s)Hc(s)(2-18)
Hold(s) = Hpe(s)Hc(s) +Hpm(s)Hc(s) (2-19)
Fourier Coefficients
MLE Model
|Hol
t(jω)|,—
10-1 100 101 10210-2
10-1
100
101
102
(a) Target open loop frequency response - magnitude.
∠(H
olt(jω)),deg
10-1 100 101 102-630
-540
-450
-360
-270
-180
-90
0
90
180
(b) Target open loop frequency response - phase.
|Hol
d(jω)|,—
10-1 100 101 10210-2
10-1
100
101
102
(c) Disturbance open loop response - magnitude.
∠(H
old(jω)),deg
10-1 100 101 102-630
-540
-450
-360
-270
-180
-90
0
90
180
(d) Disturbance open loop response - phase.
Figure 2-3: Frequency response of target and disturbance open loops for run 175 (evaluationphase), using simulated time signals.
The implementation in Simulink of the training experience conducted by Pool et al. [8]
served to produce data that allowed the validation of the algorithms developed to process the
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
14 Experiment Preparation
results of the current experiment. In this Section, the main results of identification techniques
were shown, but during the experiment preparation the results gathered with this simulation
were also useful to develop and validate other algorithms and analysis techniques typically
used in tracking experiments, like coherence determination, crossover frequency, and phase
margin estimation, decomposition of tracking error and control activity variance, and other
dependent metrics used to evaluate the results of the current experiment.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
2-2 Pilot Experiments 15
2-2 Pilot Experiments
Following the literature study developed and the offline simulations, the design of the training
experiment began. In this section, insight on crucial aspects of the experiment design is given,
justifying the most relevant design decisions made. Namely, two pilot experiments to define
the final controlled element dynamics and the outside visual scene are described and their
results analyzed.
2-2-1 Controlled Element Dynamics
The first design choice to be made for the final experiment was the dynamics to be controlled.
A pilot experiment was conducted in the SRS on March 9, 2016, where three different dy-
namics were tested, and the results processed to evaluate how these dynamics performed in
terms of controlling effort, tracking performance and human operator modeling. The task
performed is shown in Fig. 2-4. Human operators perceived the tracking error to minimize
with a compensatory display presented in the Primary Flight Display (PFD), shown in Fig.
2-5. No other cues were provided (the SRS outside visual and motion systems were off).
Human Operator
ft +
-
e
Error Response
Hpe
ue
n
+ + u
fd
++Controlled Dynamics
Hcφδ
Figure 2-4: Schematic representation of the roll tracking task.
Figure 2-5: Compensatory tracking display shown in the PFD.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
16 Experiment Preparation
The dynamics tested are described in the following paragraphs, together with a description
of the structure of the human operator control behavior model utilized. In Fig. 2-6 the
frequency response of the three dynamics tested is shown.
1. The first dynamics tested were the dynamics used in the training experiment mentioned
in Sec. 2-1. The dynamics are given in Eq. (2-1) and correspond to a reduced-order
linearized model for the elevator-to-pitch dynamics of a Cessna Citation I.
Considering the frequency response shown in Fig. 2-6, the structure for the human
operator behavior was chosen the same as the one introduced in Sec. 2-1, with the error
response model given by Eq. (2-20), being equivalent to the one used in Pool et al. [8].
This structure is chosen for the human operator model following the crossover model by
McRuer et. al (1965) [21], making the dynamics of the target open loop in the crossover
region to approximate a single integrator.
Hpe(s) = Ke(Tleads+ 1)2
Tlags+ 1e−sτeHnm(s) (2-20)
With Hnm(s) representing the neuromuscular system dynamics given in Eq. (3-4).
Hnm(s) =ω2nm
s2 + 2ζnmωnms+ ω2nm
(2-21)
Six parameters define the human operator equalization response: the error gain Ke, a
lead time constant, Tlead, and a lag time constant, Tlag. Human operator limitations
are included in a time delay τe and in the neuromuscular system, a second-order system
characterized by a natural frequency ωnm and a damping ratio ζnm.
2. The second dynamics tested correspond to the dynamics used by P. M. T. Zaal et al.
[16], from a mid-size twin-engine commercial transport aircraft with a gross weight of
185,800 lbs. The roll dynamics, given in Eq. (2-22), correspond to a linearization close
to the stall point, at an altitude of 41,000 ft and an airspeed of 150 kts.
Hc(s) = 3.91040
(
s2 + 0.2175s+ 0.5861)
(s+ 0.7599) (s− 0.02004) (s2 + 0.1133s+ 0.6375)(2-22)
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
2-2 Pilot Experiments 17
Considering the frequency response shown in Fig. 2-6, the following structure for the
human operator behavior was adapted, with the error response model given by Eq.
(2-23), being equivalent to the one used in [16].
Hpe(s) = Ke (Tleads+ 1) e−sτeHnm(s) (2-23)
From Eq. (2-23), it can be concluded that five parameters define the human operator
equalization response: the error gain Ke and a lead time constant, Tlead. Human
operator limitations are, as before, included in a time delay τe and in the neuromuscular
system, defined in the same way as for the first dynamics introduced.
3. The third dynamics that were considered in this pilot experiment were a pure double
integrator, shown in Eq. (2-24). The goal of including these dynamics in the testing set
was to experience the difficulties they create to the human operator when compared to
realistic aircraft dynamics.
Hc(s) =1
s2(2-24)
The structure used to model human operator response to the tracking error in this
dynamics was the same as used for the unstable stall-point roll dynamics given by Eq.
(2-23). This is done because the behavior of both dynamics is approximately the same
near the crossover frequency (usually between 1 — 5 rad/s) [16].
Cessna Citation IUnstable RollDouble Integrator
|Hc(jω)|,—
10-1 100 10110-2
10-1
100
101
102
(a) Controlled element dynamics — magnitude.
∠H
c(jω),deg
10-1 100 101-210
-180
-150
-120
-90
-60
(b) Controlled element dynamics — phase.
Figure 2-6: Frequency response of the three considered dynamics.
Regarding the frequency response of the dynamics considered, it can be seen in Fig. 2-6 that
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
18 Experiment Preparation
both the pitch and the unstable dynamics approach the double integrator dynamics for high
frequencies. On low frequencies, the unstable dynamics and the pitch dynamics are similar to
a single integrator, which means they will be easier to control than the double integrator. The
main difference in terms of frequency response between these two dynamics is the crossover
frequency, which is higher for the Cessna Citation I, and the phase, which is higher for the
pitch dynamics and therefore further away from the critical -180 degrees at the crossover
frequency. These two effects combined make the pitch dynamics easier to control than the
unstable dynamics.
Results
Two testing subjects performed three tracking runs with each of the previous defined dynam-
ics. The runs were averaged to mitigate measurement noise and non-linearities affecting the
runs. One subject, henceforth mentioned as subject 1, had significant experience in compen-
satory tasks and the other subject, henceforth mentioned as subject 2, had relatively little
experience in tracking tasks. The dynamics are henceforth referred to as P, U and D, mean-
ing, respectively, the Cessna Citation I elevator-to-pitch dynamics of Eq. (2-1), the unstable
stall-point roll dynamics of Eq. (2-22) and the pure double integrator dynamics of Eq. (2-24).
Tracking Error and Control Activity RMS
In Figure 2-7 the average of the root mean square value of the tracking error and control
activity time signals of the three runs performed in each condition is shown for both subjects.
It is visible the differences in task proficiency of the subjects. Both subjects performed better
with the Pitch dynamics, and the double integrator yielded the worse results. The unstable
dynamics constitute an intermediate situation between the other two considered dynamics.
Human Modeling Parameters
A time-domain identification algorithm was applied to the three-run average for each condi-
tion. The method, described in [20], consists on a genetic algorithm optimization method to
generate the initial estimation of a Gauss-Newton optimization procedure. The results ob-
tained for each subject are shown in Table 2-2. It can be seen that Subject 1 has higher gains
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
2-2 Pilot Experiments 19
Subject 1Subject 2
Dynamics
σe,deg
P U D
0
1
2
3
4
(a) Tracking error.Dynamics
σu,deg
P U D
0
1
2
3
4
(b) Control input.
Figure 2-7: Root mean square value of tracking error and control activity for both subjects inthe three dynamics tested.
in every dynamics, together with lower lead constants and higher lag constants, which is con-
sistent to what was found in terms of tracking performance. Subject 1 has approximately the
same neuromuscular frequency and damping ratio in every dynamics, while subject 2 shows
more variation, especially in the double integrator case. For these dynamics, the damping
coefficient is higher than 1 suggesting the model did not hold a proper estimation of this
neuromuscular parameter.
Table 2-2: Parameter estimates for both subjects for three dynamics tested.
Pitch Dynamics Unstable Dynamics Double Integrator
Ke, — Tlead, s Tlag, s τe, s ωnm, rad/s ζnm, — Ke, — Tlead, s τe, s ωnm, rad/s ζnm, — Ke, — Tlead, s τe, s ωnm, rad/s ζnm, —
Subject 1 3.76 0.52 1.53 0.23 10.41 0.23 1.04 0.63 0.24 10.59 0.15 0.61 0.83 0.22 10.32 0.17Subject 2 2.21 0.78 2.60 0.27 9.42 0.65 0.39 1.15 0.28 9.26 0.62 0.14 2.23 0.37 19.87 1.08
In Figures 2-8 and 2-9, the frequency response of the identified human control behavior and
the target open loop are, respectively, shown for subject 1. Considering these plots, it can be
seen again that the unstable roll dynamics are an intermediate situation between the other
two dynamics. Pilot behavior in pitch and unstable dynamics is similar, and in the open
loop it is visible that, while the crossover frequency is approximately the same, the unstable
dynamics have a lower phase margin than the pitch dynamics.
In Fig. 2-10, where crossover frequencies and phase margins are shown for both subjects, it
can be seen that on one hand the phase margin of the double integrator is small, indicating
difficulties for the human operator to successfully control the system, and on the other hand
the pitch dynamics make the task relatively easy. It can also be seen that subject 2 has a
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
20 Experiment Preparation
Cessna Citation IUnstable RollDouble Integrator
ω, rad/s
|Hpe(jω)|,—
10-1 100 101 10210-1
100
101
(a) Human operator response — magnitude.
ω, rad/s
∠(H
pe(jω)),deg
10-1 100 101-500
-400
-300
-200
-100
0
100
(b) Human operator response — phase.
Figure 2-8: Subject 1 frequency response for the three dynamics tested.
Cessna Citation IUnstable RollDouble Integrator
ω, rad/s
∣ ∣
Hpφ(jω)∣ ∣,—
10-1 100 101 10210-2
10-1
100
101
102
(a) Open loop frequency response — magnitude.
ω, rad/s
∠(
Hpφ(jω))
,deg
10-1 100 101 102-600
-500
-400
-300
-200
-100
0
(b) Open loop frequency response — phase.
Figure 2-9: Subject 1 target open loop frequency response for the three dynamics tested.
Subject 1
Subject 2
Dynamics
ωc,rad/s
P U D
0
1
2
3
4
(a) Disturbance open loop crossover frequency.Dynamics
φm,deg
P U D
20
40
60
80
100
(b) Target open loop phase margin.
Figure 2-10: Target open loop crossover frequency and phase margin for both subjects for thethree dynamics tested.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
2-2 Pilot Experiments 21
abnormally high phase margin for the pitch dynamics, but his crossover frequency is, con-
trastingly, abnormally low, being a good example on the trade-off a human operator has to
make on a tracking task between performance and stability.
Conclusions
Regarding the pilot experiment made to decide which dynamics would be used on the fi-
nal experiment setup, the following conclusions were drawn based on the results previously
presented and on the opinion of both subjects performing the pilot experiment.
• Pitch Dynamics: These dynamics are easy to control, even for task-naive partici-
pants, which means that it is unlikely that task-naive subjects loose the control of these
dynamics. This is a clear advantage of these dynamics because if that would happen
the training progress would be interrupted, and furthermore if it would happen on a
motion-base condition it would cause a simulator crash, which is something to avoid.
Furthermore, there is also background research available where these dynamics were
used, which allows for a cross-checking of the final results. However, these dynamics
might be excessively easy, failing to challenge subjects who need to perform 200 tracking
runs. Also, with easy dynamics, subjects might master them on a early stage of the
experiment, jeopardizing the exponential acquisition of manual control skills expected;
• Unstable Roll Dynamics: These dynamics, while being harder to control than the
previous ones, are also relatively easy to master, which makes it again unlikely for
subjects to loose control over these dynamics. Being slightly harder makes these dy-
namics more suitable to performance improvements throughout the 200 tracking runs
performed by each subject. The most considerable difference between these dynamics
and the pitch dynamics is the constant focus necessary to execute the control tracking
task with these dynamics, given their unstable characteristic. Background research done
with these dynamics also provides a safety net for cross-checking final results, however
not in the same degree as the previous dynamics do, with whom training experiments
with task-naive subjects have already been performed;
• Double Integrator Dynamics: Given the marginal stability of the double integrator,
these are the hardest dynamics to control. They require a permanent focus of the
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
22 Experiment Preparation
participant, which is an advantage as participants would need to be extremely focused
throughout the entire experiment, but it can also increase fatigue effects on the subjects.
However, given the naivety of the subjects who will be performing the experiment, these
dynamics might cause problems in motion conditions where simulator limits are easily
attained.
Given the aforementioned arguments, which were carefully weighted and evaluated, the final
decision was to utilize in this training experiment the unstable dynamics given in Eq. (2-22),
as they offer a piece of both worlds, being challenging enough without being excessively hard
to control.
2-2-2 Outside Visual Cues
The second main choice to be made regarding the final experiment setup was the outside
scenario to provide out-of-the-window visual cues to the human operator. Again, three options
were considered for the outside visual scene and a pilot experiment was performed on the
SRS to evaluate what outside visual scenario best fitted the experiment purpose. This pilot
experiment was performed on April 28, 2016, and consisted on a tracking task similar to the
one described in the previous section, with the addition of a feedback path to the controlled
dynamics, which could be provided by motion and/or visual cues, defining four conditions
(C1, C2, C3 and C4) as explained in Table 2-3, with the tracking error being presented in
the PFD in the form of a compensatory display for every condition.
Table 2-3: Experimental conditions for the outside visual cues pilot experiment.
PFD Outside Visuals Motion
C1 On Off OffC2 On On OffC3 On Off OnC4 On On On
The dynamics utilized were the ones chosen in Section 2-2-1, the unstable stall-point roll
dynamics given in Eq. (2-22). The tracking task block diagram is given in Fig. 2-11, where
the dashed channel represents the roll response which is available when motion and/or visual
cues were provided, meaning that in C1 only the error response channel is considered in
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
2-2 Pilot Experiments 23
the human operator model. Therefore in this pilot experiment a double-channel structure
was considered for conditions C2, C3 and C4, where the error response was modeled in the
same way as in the previous pilot experiment (Eq. (2-23)), and the response to the roll
feedback (provided by the outside visual scene and/or the physical motion of the simulator)
was modeled with the structure given in Eq. (2-25).
Hpφ(s) = sKφe−τφsHnm(s) (2-25)
Human Operator
ft +
-
e
Error Response
Hpe
Roll Response
Hpφφ
ue
n
uφ
-
+ + u
fd
++Controlled Dynamics
Hcφδ
Figure 2-11: Schematic representation of the roll tracking task.
A description of the three scenarios considered is given in the following paragraphs.
• Vertically moving checkerboard patterns: This outside visual scenario provides
peripheral visual cues using two vertically moving checkerboards positioned in the lateral
windows of the simulator’s cockpit. The checkerboards move vertically according to the
roll angle, providing human operators with roll rate information without giving a roll
angle reference. While being a peripheral visual cue and not a fully outside visual cue,
the checkerboards are reported to give strong roll inputs, as peripheral visual cues are
known to be primarily important in visual motion perception [22], and were already
used in roll compensatory tracking tasks [13]. In Fig. 2-12(a) one checkerboard panel
is displayed, and in Fig. 2-12(b) the visual on the simulator cockpit is shown.
• Realistic outside environment: This outside visual scenario consists on a realistic
environment corresponding to Schiphol Airport, in Amsterdam, the Netherlands. The
center projection of the visual scene presented to the human operator is shown in Fig.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
24 Experiment Preparation
(a) Right checkerboard. (b) Simulator cockpit.
Figure 2-12: Checkerboard visual scene.
2-13(a), with the simulator cockpit in 2-13(b). When considering a realistic outside
environment to provide roll cues, some aspects were taken into account. If a realistic
situation is adopted, with the simulated aircraft flying at a certain altitude, then the
horizon line is barely visible from the pilots’ position, thus resulting in poor roll rate
feedback information and in a weak roll sensation when motion is not provided. To
increase the available visual information, elements like clouds could be added to the
visual scenario, or the altitude could be lowered, so that the horizon line, buildings, trees
and other elements of Earth surface are visible to the pilot, providing the needed roll
information. The last solution was adopted, thus the outside environment presented to
human operators consisted on a fixed 5 meter height perspective, that will roll following
the controlled element dynamics. The reduced height causes a perception of the roll
rotation based on the position of the horizon line and its relative position with respect
to the buildings and other visible elements.
• Flow-field environment: This outside visual scenario gives information on the dy-
namics exploring flow-field perception. It consists on a black background with white
dots randomly distributed in a simulated 3-dimensional space, projected in the windows
of the simulator cockpit as shown in 2-14(b) and simulates a star field, shown in Fig.
2-14(a). This scenario is based on the optic flow concept. When the simulation starts,
a radial optic flow pattern emerges, created by the stars through two types of simulta-
neous movement, a translational and a rotational movement, made with respect to the
central point of the projected image, the focus of radial outflow [23]. To understand how
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
2-2 Pilot Experiments 25
(a) Realistic visual scene. (b) Simulator cockpit.
Figure 2-13: Realistic visual scene.
(a) Flow-field scene. (b) Simulator cockpit.
Figure 2-14: Flow-field visual scene.
the projection is generated, it helps if the human operator moving through the world
is modeled using a camera analogy. The translational movement is simulated with a
constant forward speed of the camera, and the rotational movement is simulated with
the rotation of the camera around an axis perpendicular to the perspective according
to the roll angle of the controlled element. The combination of both aspects provides
a strong visual perception of the controlled dynamics. To the described visual scenario
a explicit compensatory display showing the tracking error was added, overlapped in
the visual scenario, repeating the information provided in the PFD. This was done so
that a reference for the null error was explicitly given, because if the null position is
only provided by the PFD then this visual cue would become peripheral (the human
operator would have to look down to perform the task).
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
26 Experiment Preparation
Results
The same two test subjects who performed the dynamics pilot experiment performed the
task in the four conditions for the three outside visual scenes, making a total of 12 runs per
subject. The results are subsequently described.
Tracking Error and Control Activity RMS
In Figure 2-15, the RMS of the tracking error and the control input are shown for both
subjects, in the four cues combinations and the three outside scenarios. The difference in
performance of both subjects is again clear as all the red lines in Fig. 2-15(a) are above the
blue lines. To compare visual scenarios, conditions 2 and 4 should be considered, because
in these conditions the outside visuals are On. For Subject 1, the best performances are
attained with the checkerboards and the worse with the flow-field. For Subject 2 the best
performances are attained with the realistic environment, while the checkerboards and the
flow-field register similar performances.
Subject 1Subject 2CheckerboardsRealisticFlow Field
σe,deg
C1 C2 C3 C4
0
1
2
3
4
(a) Tracking error RMS.
σu,deg
C1 C2 C3 C4
0
1
2
3
4
5
(b) Control input RMS.
Figure 2-15: RMS of signals e and u for both subjects in the three outside visuals tested.
Human Operator Modeling Results
The 24 runs performed were submitted to a maximum likelihood time domain identification
procedure in order to estimate the parameters defining the error response Hpe and the roll
response Hpφ of the human operator. Each run was individually identified, thus without
performing any between-runs averaging, degrading the quality of the estimates. The results
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
2-2 Pilot Experiments 27
of the human operator control behavior in condition 2 (with the outside visual scene on and
the physical motion cues off) will be carefully analyzed, because in this condition outside
visual cues are mostly relevant to the creation of the roll feedback channel.
In Fig. 2-16, Subject 1 error and roll frequency response plots are shown, for the three outside
visuals provided, in condition C2. Given that only outside visual cues were provided in C2,
they are exclusively responsible for the creation of the roll response. It is visible in Fig.
2-16(c) that the checkerboards provide the strongest roll feedback channel. The flow-field
fails to create effectively this feedback channel. If open loop frequency response is considered,
shown in Figure 2-17, it is clear that with the checkerboards this human operator registered
the best open-loop characteristics, and therefore with the checkerboards the roll feedback
channel is the most effective from the three tested visuals.
CheckerboardsRealisticFlow Field
ω, rad/s
|Hpe(jω)|,—
10-1 100 101 10210-2
10-1
100
101
102
(a) Error response - magnitude.
ω, rad/s
∠(H
pe(jω)),deg
10-1 100 101 102-630
-540
-450
-360
-270
-180
-90
0
90
180
(b) Error response - phase.
ω, rad/s
∣ ∣
Hpφ(jω)∣ ∣,—
10-1 100 101 10210-2
10-1
100
101
102
(c) Roll response - magnitude.
ω, rad/s
∠(
Hpφ(jω))
,deg
10-1 100 101 102-630
-540
-450
-360
-270
-180
-90
0
90
180
(d) Roll response - phase.
Figure 2-16: Frequency response of Subject 1 for C2 (Visuals On, Motion Off), comparing thethree outside visual scenarios.
Crossover Frequencies and Phase Margins
In Figure 2-18 the disturbance and target crossover frequencies and phase margins for both
subjects are shown. Looking at C2 from subject 1, it is seen that checkerboards cause a
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
28 Experiment Preparation
CheckerboardsRealisticFlow Field
replacemen
ω, rad/s
|Hol
d(jω)|,—
10-1 100 101 10210-2
10-1
100
101
102
(a) Disturbance open loop - magnitude.
ω, rad/s
∠(H
old(jω)),deg
10-1 100 101 102-630
-540
-450
-360
-270
-180
-90
0
(b) Disturbance open loop - phase.
ω, rad/s
|Hol
t(jω)|,—
10-1 100 101 10210-2
10-1
100
101
102
(c) Target open loop - magnitude.
ω, rad/s
∠(H
olt(jω)),deg
10-1 100 101 102-630
-540
-450
-360
-270
-180
-90
0
(d) Target open loop - phase.
Figure 2-17: Open loop frequency response of Subject 1 for C2 (Visuals On, Motion Off),comparing the three outside visual scenarios.
better performance (higher crossover frequencies and lower phase margins). For subject 2,
the same is visible in target crossover frequency and both phase margins, while no clear
difference between the scenarios exists in disturbance crossover frequency. In C2 and for both
subjects, the flow-field visual scenario yields the worse results in these parameters, with the
realistic environment yielding intermediate results. Furthermore, motion is seen to decrease
target crossover frequencies and increase disturbance crossover frequencies and target phase
margins. No clear effect on disturbance phase margin is seen to be introduced by motion.
Conclusions
Considering the pilot experiment designed to decide which outside visual scene would be uti-
lized as out-of-the-window visual cue in this training experiment, the following considerations
were drawn based on the results presented in this Section and on the opinion of both subjects
performing the test session.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
2-2 Pilot Experiments 29
wc d,rad/s
C1 C2 C3 C4
0
1
2
3
4
(a) Disturbance crossover frequency.
Subject 1Subject 2CheckerboardsRealisticFlow Field
φd,deg
C1 C2 C3 C4
0
25
50
75
100
(b) Disturbance phase margin.
wc t,rad/s
C1 C2 C3 C4
0
1
2
3
4
(c) Target crossover frequency.
φt,deg
C1 C2 C3 C4
0
25
50
75
100
(d) Target phase margin.
Figure 2-18: Crossover frequencies and phase margins for both subjects in the three outsidevisual scenes tested.
• Vertically moving checkerboard patterns: This out-of-the-window visual cue has
the main advantage of providing a strong visual roll rotation input with relative sim-
plicity. Another advantage is the fact that it was already used in previous roll-tracking
tasks, with its effects being well-known and described. It is not however a full out-of-the-
window visual cue, as it provides only peripheral visual cues. This fact is nonetheless
not problematic because it is known that peripheral visual cues are the main responsi-
ble for the visual perception of motion, and subjects performing tracking experiments
with these visuals reported that checkerboards made them feel like if they were actually
moving.[13] This translated to the results obtained in this test session, with the verti-
cally moving checkerboards proving to be clearly more effective in giving roll motion
feedback than the other visual cues. This is visible in lower tracking error RMS, higher
gains in the roll feedback response and higher crossover frequencies attained with the
checkerboards as visual cue. The fact that this cue is not realistic may constitute a
disadvantage in terms of transferring the results of the experiment to real flight sim-
ulators, but on one hand the entire experiment has a fundamental character typical
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
30 Experiment Preparation
in academic research, and on the other hand, the remaining solutions do not offer a
significant increase in terms of realism. Another positive point of these dynamics is the
fact that they are highly unlikely to create motion sickness.
• Realistic outside environment: This visual setup is the visual cue which is clos-
est to reality, but the fact that the aircraft was simulated to be in low altitude and
not moving forward lowers the realism of this scenario. Subjects also reported some
sickness effects caused by this visual scene. Looking at the results obtained when the
Schiphol Airport environment was given as roll cue, it was seen that these cues provide
a strong visual input, with good results in tracking performance and human operator
modeling results. Another effect described by the subjects was the fact that it was not
easy to steer the dynamics to the null error, which is explained by the fact that these
dynamics provide roll-attitude reference, as the zero error corresponds to the horizon
being aligned. Therefore subjects would try to minimize the error without looking at
the PFD, which leads to worse performance because perceiving zero error is easier with
the explicit compensatory display.
• Flow-field environment: This out-of-the-window visual cue was the less effective in
providing a strong roll rotation perception, registering the worse tracking performances,
an effect also seen in human operator modeling results. Subjects described some motion
sickness caused by the flow-field environment. They also reported that the perception
of the zero-error position was harder with this cue, because they would not look at the
explicit display in the PFD but to the explicit display which was overlapped into the
visuals, where it was harder to perceive the null position, possibly due to the interaction
with the visual background. This scene was developed to take advantage of optic flow
concepts, which were previously applied to tracking tasks but not in a roll-axis tracking
task. The results of this test session seem to prove that this concept fails to be effective
as a roll cue, but a more extensive study, with perhaps a different concretization of the
optic flow concept, would be beneficial to further validate this conclusion.
Considering the results presented, together with the opinions collected from the subjects and
the arguments described, the conclusion of this test session was to use the checkerboards as
the outside visual cue in this quasi-transfer-of-training experiment.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
Chapter 3
Experimental Methods
Following the experiment design decisions given in the previous Chapter, the methods followed
and the full experiment design are given in this Chapter, with a description of the control
task, the experiment setup, the participants and the apparatus, followed by a description on
the way the experimental data was analyzed. This Chapter ends with the main hypothesis
to be tested.
3-1 Control Task
This human-in-the-loop training experiment considered a compensatory roll-axis tracking
task, following the same procedures as various earlier investigations on skill-based manual
control training [8, 15, 16, 13]. In Figure 3-1 this task is schematically represented. The
human controller was asked to follow a target roll angle, specified by the tracking signal ft,
as accurately as possible. Simultaneously, the human operator had to reject disturbances on
the controlled system Hc, which were induced by the disturbance signal fd. This disturbance
signal was summed to the human operator’s input, u, and directly affected the controlled
dynamics. In order to identify and model the multi-channel human operator response, char-
acterized by Hpe and Hpφ , the disturbance signal fd and the target signal fd were independent
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
32 Experimental Methods
sum-of-sines signals [24, 19, 20]. Given the quasi-linear human operator model used, the con-
trol input had contributions from the error response, ue, the roll response, uφ, and a remnant
n accounting for nonlinear behavior and measurement noise.
Human Operator
ft +
-
e
Error Response
Hpe
Roll Response
Hpφφ
ue
n
uφ
-
+ + u
fd
++Controlled Dynamics
Hcφδ
Figure 3-1: Schematic representation of the roll tracking task.
In the experiment (see Fig. 3-2(a)), the human operator perceived the dynamics being con-
trolled with a compensatory display which resembled a basic Primary Flight Display (PFD),
shown in Fig. 3-2(b). This compensatory display showed the deviation e between the current
aircraft roll angle, φ, and the target roll angle, ft.
The out-of-the-window visual cues used were chosen with a pilot experiment described in
Section 2-2-2 and are based on various compensatory roll-axis tracking experiments [25, 1,
26, 13]. They consisted on two vertically moving checkerboard panels (see Fig. 3-2(c)),
providing a strong roll motion sensation without giving reference of roll-attitude. In the
moving base condition pure roll motion was provided, thus without any washout or lateral
specific force compensation.
(a) Simulator Cockpit.
e
(b) Central Display. (c) Out-of-the-window visuals.
Figure 3-2: Simulator cockpit, central display and the out-of-the-window scene.
The controlled dynamics Hc were also chosen with a pilot experiment, described in 2-2-1.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
3-1 Control Task 33
ω, rad/s
|Hc(jω)|,—
10-1 100 10110-2
10-1
100
101
102
ω, rad/s
∠H
c(jω),
deg
10-1 100 101-180
-150
-120
-90
Figure 3-3: Frequency response of the controlled element dynamics.
They are the dynamics used in Ref. [16] presented in Eq. (3-1), multiplied by a gain of
5. These dynamics correspond to a mid-size twin-engine commercial transport aircraft with
a gross weight of 185,800 lbs, linearized in a flight condition close to the stall point, at an
altitude of 41,000 ft and an airspeed of 150 kts.
Hc(s) = 3.91040
(
s2 + 0.2175s+ 0.5861)
(s+ 0.7599) (s− 0.02004) (s2 + 0.1133s+ 0.6375)(3-1)
An obvious remark regarding the transfer function presented in Eq. (3-1) is that the roll
dynamics of the aircraft are unstable, due to the spiral mode. These dynamics approximate
a single integrator(
1s
)
at low frequencies (< 0.75 rad/s) and a double integrator(
1s2
)
at
frequencies higher than 0.75 rad/s, as seen in Fig. 3-3, due to the effect of the pole at
s = -0.7599 rad/s.
The human operator control behavior was modeled in this compensatory tracking task using
a quasi-linear model [21, 27]. As shown in Fig. 3-1, the output of the human operator, the
control input signal u, is the sum of a linear response and a remnant signal n. The linear
response has two contributions, Hpe and Hpφ , which respectively model the response to the
roll tracking error (available from the PFD) and the response to the roll feedback (available
via the out-of-the-window cues or simulator’s motion) [28, 29]. The remnant signal accounts
for measurement noise and nonlinearities which are not described by the linear response
functions. Determining the form of the transfer functions Hpe and Hpφ and the evolution of
their parameters throughout the progress of a training experiment has proven to be of great
help in understanding and quantifying the learning process of both visual and motion cues
by the initially task-naive participants [16, 30].
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
34 Experimental Methods
3-2 Forcing Functions
The disturbance and target forcing functions, fd and ft, were independent sum-of-sines signals
defined by Eq. (3-2), as used in a number of previous tracking studies [19, 24, 31].
fd,t(t) =
Nd,t∑
k=1
Ad,t[k] sin (ωd,t[k]t+ φd,t[k]) (3-2)
In Eq. (3-2), Ad,t[k], ωd,t[k] and φd,t(k) respectively indicate the amplitude, frequency and
phase of the k-th sine in the forcing function. Nd,t is the number of sine waves constituting
the forcing function. In this experiment, both the disturbance and target forcing functions
were the sum of Nd,t = 20 individual sinusoids, each with different amplitudes, frequencies
and phases. In Table 3-1 a list of all the parameters used to generate the forcing functions
can be found.
The frequencies of the sinusoids, ωd,t[k], were purposely defined as integer multiples of the
measurement time base frequency, ωd,t[k] = nd,t[k] ωm, where ωm = 2π/Tm = 0.0767 rad/s
and Tm = 213/fs = 81.92 s is the measurement time, counting from the end of the run (each
run had 95 seconds and only the last 81.92 seconds were used, using a sampling frequency of
fs=100 Hz). The selected integer multiples were used in previous studies [8], guaranteeing
the twenty sinusoid frequencies covered the frequency range of human control at regular
intervals on a logarithmic scale. Moreover, the integer multiples were chosen in consecutive
pairs, creating double bands of input frequencies so that coherence between the control signal
produced by the human operator u and the forcing functions fd and ft could be calculated,
which allowed the evaluation of human control linearity in each individual run [32].
A second-order low-pass filter was used to calculate the amplitudes of the individual sines, in
similarity with previous studies [8, 31]. The purpose of this second-order filter was to reduce
the magnitude of the amplitudes at higher frequencies, which results in a not extremely
difficult tracking task. Also matching earlier experiments [8, 31], the amplitude distributions
were scaled so that the target forcing function power was 25% of that of the disturbance
forcing function, which made this task predominantly a disturbance-rejection task.
Five realizations of fd and ft were used, differing only by their initial sinusoidal phases φd,t of
the individual sine-components. The set of phases chosen yielded signals with a Gaussian-like
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
3-3 Experiment Setup 35
Table 3-1: Experiment forcing function (disturbance and target) data
Disturbance signal, fd Target signal, ftnd ωd, rad/s Ad, deg φd,1, rad φd,2, rad φd,3, rad φd,4, rad φd,5, rad nt ωt, rad At, deg φt,1, rad φt,2, rad φt,3, rad φt,4, rad φt,5, rad2 0,15 0,03 6,28 3,52 2,64 2,83 2,96 5 0,38 0,51 5,68 3,99 3,92 6,00 4,393 0,23 0,05 1,25 5,41 2,07 5,64 1,00 6 0,46 0,49 0,83 4,35 1,42 5,23 5,779 0,69 0,13 1,46 2,16 5,64 5,18 4,57 13 1,00 0,34 0,54 5,35 5,17 4,75 4,9310 0,77 0,09 5,45 0,11 3,78 5,70 1,51 14 1,07 0,33 1,14 5,92 1,57 6,28 4,2322 1,69 0,27 1,96 0,66 4,94 4,09 1,81 27 2,07 0,16 2,93 3,84 4,05 2,88 4,0123 1,76 0,28 3,29 2,88 0,38 4,65 4,04 28 2,15 0,15 2,83 3,48 6,24 1,23 2,3936 2,76 0,36 2,62 1,96 1,88 4,43 3,33 41 3,14 0,08 6,02 4,99 4,04 1,21 2,9137 2,84 0,36 4,28 5,61 4,38 4,98 3,45 42 3,22 0,08 1,74 4,97 2,71 0,29 0,4649 3,76 0,40 6,28 5,63 0,73 3,80 4,31 53 4,07 0,06 3,90 4,26 1,71 1,88 2,5650 3,83 0,41 3,93 0,31 2,07 2,35 5,67 54 4,14 0,05 0,74 1,00 4,03 4,62 2,0869 5,29 0,47 4,39 0,65 4,09 3,60 1,54 73 5,60 0,03 5,65 4,69 0,67 0,89 4,5670 5,37 0,48 0,65 1,14 4,42 4,20 1,80 74 5,68 0,03 3,70 5,01 5,85 0,97 3,3397 7,44 0,59 6,17 0,35 2,82 0,34 6,24 103 7,90 0,02 3,63 5,44 5,13 0,74 1,4399 7,59 0,60 2,17 2,92 2,69 5,85 0,84 104 7,98 0,02 1,42 5,78 4,14 5,81 1,44135 10,35 0,79 3,80 2,27 4,91 4,54 3,25 139 10,66 0,02 3,64 0,86 2,66 5,21 5,97136 10,43 0,80 2,58 2,57 4,31 2,20 1,43 140 10,74 0,02 5,94 2,05 5,65 1,08 4,55169 12,96 1,03 2,84 6,07 4,46 6,16 4,57 193 14,80 0,01 4,74 2,03 4,28 0,70 5,76170 13,04 1,04 4,11 2,98 4,22 3,94 0,44 194 14,88 0,01 3,72 2,88 1,92 2,81 5,28224 17,18 1,53 4,98 0,77 4,69 2,20 3,57 229 17,56 0,01 4,32 3,01 1,03 4,44 2,67225 17,26 1,54 5,67 3,90 4,70 4,16 5,20 230 17,64 0,01 2,15 2,93 3,21 0,53 2,50
distribution and an average crest factor [33]. This was done in order to prevent subjects
from recognizing parts of the signals because of the repeated exposure inherent to performing
200 tracking runs. With a combination of five disturbance signals and five target signals,
randomly assigned to consecutive tracking runs following a latin-square distribution, it was
virtually impossible for the subjects to memorize the signals.
The disturbance signal was preshaped by the inverse of the aircraft dynamics to compensate
for the fact that it is actually inserted before Hc (see Fig. 3-1).
3-3 Experiment Setup
The experiment was divided in two phases, henceforth referred to as training and evaluation.
During the training phase, the task-naive participants were trained in the roll tracking task
previously described until their level of task performance stabilized. They were subsequently
transferred to the evaluation phase where the same roll tracking task was performed with
different cues being provided.
Participants were divided in two experimental groups, as shown in Fig. 3-4. The first group,
henceforth referred to as Group NV, was trained with only the PFD, thus without neither
out-of-the-window visuals or motion cues (NV NM), and they were transferred to evaluation
conditions where they had access to the PFD and motion feedback (NV M). The second
group, henceforth referred to as Group V, was trained without motion but with the PFD
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
36 Experimental Methods
and the out-of-the-window visual cues (V NM), and it was transferred to the same evaluation
configuration as the Group NV where the PFD and motion cues were available, without
peripheral visual cues (NV M). Therefore, the only difference between the two experimental
groups was the presence of out-of-the-window visual cues in the training phase.
Day 1
Session 1
25 runs
V NM
Session 2
25 runs
V NM
Day 2
Session 3
25 runs
V NM
Session 4
Transfer
25 runs
V NM
Day 3
Session 5
25 runs
NV M
Session 6
25 runs
NV M
Day 4
Session 7
25 runs
NV M
Session 8
25 runs
NV M
Visuals Group - Group V
Training Evaluation
Day 1
Session 1
25 runs
NV NM
Session 2
25 runs
NV NM
Day 2
Session 3
25 runs
NV NM
Session 4
Transfer
25 runs
NV NM
Day 3
Session 5
25 runs
NV M
Session 6
25 runs
NV M
Day 4
Session 7
25 runs
NV M
Session 8
25 runs
NV M
No - Visuals Group - Group NV
Training Evaluation
Figure 3-4: Quasi-transfer-of-training experiment design.
Each phase of the experiment consisted of a fixed number of 100 tracking runs, therefore each
subject performed 200 runs in total. The 95-second runs were performed in eight sessions of 25
runs each. The eight sessions were performed in four consecutive days, therefore two sessions
on each day, with a 20 minutes break between sessions (subjects left the simulator between
sessions). This experimental configuration allowed convergence of manual control skills in
both experimental phases, with a consolidation of the acquired control skills in between days
happening outside the simulator, an effect known as offline learning [34], and it also respected
the optimum retention time between training sessions of 24 hours [35].
The experiment was performed during five weeks on four consecutive working days of each
week. Two subjects performed their two daily sessions in the morning and two subjects
performed their two daily sessions in the afternoon, meaning a total of four subjects performed
the experiment in each week. To guarantee the balancing between subjects in the groups,
every subject in one week was placed in the same group. Therefore, two weeks had subjects
from Group NV, two weeks had subjects from Group V, and the fifth week had subjects from
both groups.
In the end of each tracking run, the researcher informed the subject of their score in that run
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
3-4 Apparatus 37
(the score being the root mean square value of the tracking error signal) and asked if they
were ready for the following run. In case of an affirmative answer, the next run would start.
Otherwise, some seconds could be taken as a small break in between runs to assure subjects’
concentration levels were high and as constant throughout the experiment as possible.
3-4 Apparatus
Figure 3-5: The SIMONA Research Simulator at TU Delft.
The quasi-transfer-of-training experiment was performed in the SIMONA Research Simulator
(SRS) at the Aerospace Engineering Faculty at Delft University of Technology. Both SRS
motion and outside visual systems were used, depending on the phase of the experiment.
The SRS motion system is a hexapod with hydraulic actuators (Fig. 3-5), providing a six
degrees-of-freedom hydraulic motion system which reproduces the aircraft’s motion with a
time delay of 30 ms [36]. Given that the task performed was a pure roll tracking task, only
SIMONA roll rotation was utilized. The SRS workspace in terms of roll rotation is ±25.9◦,
and in this experiment no roll motion filter was used, thus the controlled roll attitude φ was
given one-to-one, without washout filtering. The visual system of the SRS consisted on two
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
38 Experimental Methods
projectors generating the two checkerboard panels on the left and right window views of the
simulator cockpit. The visual system delay is approximately 30 ms and all displays were ran
at a 60 Hz refresh rate [37].
Participants sat in the simulator co-pilot seat and used a right-hand sidestick to control the roll
rotation of the aircraft. The pitch axis of the sidestick was locked so that only roll commands
could be given. The active stick was set to have a linear force-deflection characteristic of 0.75
N/deg without breakout-force. The simulator rolling motion was performed around an axis
aligned with the center of the right pilot seat to ensure pure roll cueing. The compensatory
display was located in front of the participant’s seat.
Each subject adjusted both the height of the seat and the distance of the seat to the sidestick to
ensure their comfort and the correct positioning of the checkerboards in their peripheral field
of vision. These seat settings were kept constant for each subject throughout the experiment.
Furthermore, participants used noise-canceling headphones with a masking aircraft engine
sound to fully hide the acoustic noise caused by the actuators of SRS motion system.
The SRS data recording system was set to work on a sampling frequency of 100 Hz, thus
yielding 9501 points of data per variable recorded for each tracking run.
3-5 Participants
To perform this training experiment, subjects could neither have any type of previous piloting
experience nor have participated in earlier tracking experiments. Another requirement the
subjects had to comply with was to be right-handed. An initial group of twenty fully task-
naive subjects, who gave their written consent to participate in this study, performed the
experiment. A total of ten subjects were included in each group, considering people between
18 and 23 years old, with nine different nationalities and three females (two placed in Group
NV and one in Group V). Two more subjects were tested in the experiment set, but due
to performance inconsistencies were omitted from the final data set. Each subject chose a
time slot (8 sessions of 1 hour each spread over four consecutive working days) and this
defined automatically the group in which they were inserted. Therefore no explicit selection
or distribution of the subjects between the experimental groups was made.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
3-6 Data Analysis 39
3-6 Data Analysis
To understand how human operators acquire control skills throughout their learning process,
the data collected in the SRS was processed using a variety of mathematical tools and methods
which are carefully discussed in the following subsections.
3-6-1 Human Operator Modeling
Human operator control behavior was modeled and identified in each tracking run using a
multimodal quasi-linear operator model. Namely, the defining parameters of both the error
response transfer function Hpe and the roll feedback response transfer function Hpφ were
determined using identification and optimization algorithms. The models used for Hpe and
Hpφ were successfully used in earlier studies [12, 16, 31].
The considered model for the human operator error response Hpe is given by:
Hpe(s) = Ke (Tleads+ 1) e−τesHnm(s) (3-3)
With Hnm being the neuromuscular dynamics modeled by:
Hnm(s) =ω2nm
s2 + 2ζnmωnms+ ω2nm
(3-4)
For this roll tracking task with the considered controlled element dynamics, the error response
model included a gain Ke, a lead equalization term Tlead, a human operator response delay
τe, and the neuromuscular dynamics Hnm, modeled as a second-order mass-spring-damper
system with a neuromuscular frequency ωnm and a neuromuscular damping ratio ζnm. The
considered structure of Hpe is explained by the fact that human operators needed to generate
lead, because the controlled dynamics approximated a double integrator in the frequency
range where the human operator crossover frequency was expected to be for compensatory
tracking (1 - 5 rad/s) [16].
The human operator roll response Hpφ is modeled by:
Hpφ(s) = sKφe−τφsHnm(s) (3-5)
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
40 Experimental Methods
The roll response included a pure derivative term and an equalization gain Kφ, modeling
human operator limitations with a roll response delay τφ and the neuromuscular system,
modeled as in Hpe . It is convenient to mention that Hpφ characterized the sum of multiple
and separate feedback channels, namely the ones related with motion feedback, i.e., angular
accelerations detected by the semicircular canals, linear accelerations detected by the otoliths,
and motion cues from the somatosensory system [16]. For similar tasks, this model structure
has been successfully applied in earlier investigations [38]. The same human operator model
was used for experimental conditions in which either motion or out-of-the-window cues were
available. This allowed a direct assessment of how well out-of-the-window visual cues can
replace motion cues.
The multi-channel pilot model defined in Eqs. (3-3) to (3-5) contained seven free parame-
ters (Ke, Tlead, τe, Kφ, τφ, ωnm, and ζnm) which were estimated using maximum likelihood
time-domain parameter estimation techniques, as described in Section 2-1-3, on the collected
experimental data (the time-domain signals e, φ, and u). Obviously, in the training phase of
Group NV, only Hpe was fitted, as no out-of-the-window visual or motion cues were avail-
able. Firstly, ten repetitions of a genetic algorithm optimization were performed in order
to obtain ten initial rough estimates of the parameters, which were the starting point of a
Gauss-Newton optimization algorithm, yielding ten estimates for the set of parameters. The
estimate yielding the lowest value of the likelihood function was selected as the one describing
the control activity of the human operator in that run. If the lowest likelihood solution failed
to satisfy the physical restrictions inherent to the model (neuromuscular frequency between 0
and 30 rad/s and neuromuscular damping ratio between 0 and 1), another solution from the
set of ten Gauss-Newton estimates was considered. If none of the Gauss-Newton estimates
was in the domain of the model parameters, the genetic algorithm solution holding the lowest
likelihood was considered as the identified model of that run, with the validity of this lower
likelihood solution being carefully analyzed. Should this model describe the human operator
control behavior with an unacceptable low quality, the respective run would be omitted from
the final data set. This procedure was applied to the 200 runs in the training and evaluation
phases of each of the twenty subjects who performed the experiment, and from the set of 4000
tracking runs that compose the experiment, three were omitted from the final data set.
In every tracking run, the identified model Variance Accounted For (VAF) was calculated with
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
3-6 Data Analysis 41
Eq. (3-6) as a measure of the human operator model accuracy in describing the measured
control signal. The VAF is expressed in percentage and the higher the value, the better
the model describes what was exactly done by the human operator control behavior. It is
an usual practice in cybernetic studies to average the time signals over a certain number of
runs, to attenuate noise in the measured data and thus improving the model quality [31].
However, given the need to evaluate acquisition of control skills throughout individual runs,
averaging was not an option and therefore slightly lower VAF values were obtained due to
higher noise levels. Nevertheless, runs with abnormally low model VAFs (lower than 40 %)
were considered as identification outliers and excluded from the final data set considered, so
that they would not influence the group average results shown in Section III. A total of 56
runs were excluded from the final data set due to this reason, which is a reduced percentage
of excluded runs (1.4 %) for a training experiment with task-naive participants [8].
VAF =
(
1−
∑Ni=1 (u(t)− u(t))2∑N
i=1 u2(t)
)
× 100% (3-6)
The validity of the final model estimates was further analyzed with independently estimated
frequency-domain describing functions for Hpe and Hpφ , obtained with a black-box multiloop
identification method based on Fourier coefficients, as described in Section 2-1-2.
3-6-2 Other Dependent Variables
To further evaluate the progress of every subject and the effectiveness of training cues in the
evaluation setting, several additional dependent variables were calculated from the collected
data. Operator control activity and performance were evaluated, in the time domain, with the
variance of the control input, σ2u, and the roll tracking error, σ2
e , respectively, in accordance
to previous training studies [8, 16]. A decomposition on the sources (disturbance, target or
remnant) of these variances was also made. In the frequency domain, pilot performance in
rejecting the disturbance forcing function and following the target signal was determined by
the crossover frequencies and phase margins of the disturbance and target open-loop dynamics,
respectively [39]. The disturbance and target open-loop dynamics are given by (see Fig. 3-1):
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
42 Experimental Methods
Hol,d(s) = −u(s)
δ(s)=[
Hpe(s) +Hpφ(s)]
Hc(s) (3-7)
Hol,t(s) =φ(s)
E(s)=
Hpe(s)Hc(s)
1 +Hpφ(s)Hc(s)(3-8)
The disturbance and target crossover frequencies ωc,d and ωc,t are the frequencies where the
magnitude of the disturbance and target open-loop frequency responses is unity (0 dB). The
phase differences from -180 degrees at these crossover frequencies are the corresponding phase
margins φm,d and φm,t.
Finally, the coherence of the operators’ control input with respect to the disturbance and
target forcing signals was also estimated according to Eq. (3-9).
Γd,t (ωd,t [k]) =
√
√
√
√
√
∣
∣
∣Sfd,t (jωd,t [k])
∣
∣
∣
2
Sfd,tfd,t (jωd,t [k]) Suu (jωd,t [k])(3-9)
In Eq. (3-9), S (jωd,t [k]) represents the auto or cross power spectral density (PSD) function
of the signal. In each pair of averaged frequencies, ωd,t, the estimated coherence expresses
the linearity with respect to fd,t as a value between zero and one, where one means perfectly
linear control behavior [32].
3-6-3 Learning Curve Modeling
To give a quantitative insight on how operators’ control behavior changed during training
and after transfer to the evaluation condition, exponential learning curves were fitted to the
dependent measures described above. The learning curve model used is given by Eq. (3-10):
ylc(n) = pa + (1− F )n(p0 − pa) (3-10)
The exponential learning curve model in Eq. (3-10) is determined by the initial value p0, the
asymptotic value pa, and the learning rate F [6]. These parameters were calculated using
a non-linear optimization procedure using MatLab’s fminsearch to minimize the summed
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
3-7 Hypotheses 43
squared error between the experimental data yp and the learning curve model ylc, given in
Eq. (3-11). Each dependent variable had two learning curves associated, one for the training
phase and one for the evaluation. Pearson’s linear correlation coefficient ρ was determined to
evaluate fitness quality.
J =1
N
N∑
n=1
(yp(n)− ylc(n|p0, pa, F ))2 (3-11)
3-6-4 Statistical Analysis
A statistical analysis of the dependent data σ2e and σ2
u was performed to compare different
phases of the experiment. Three pairwise comparisons (dependent t-tests) were done for
each variable, corresponding to the expected evolutions throughout the experiment. Training
comparison took the average of each subject on runs 1-5 and 96-100. Transfer comparison
considered the average of each subject on runs 96-100 and 101-105. Evaluation comparison
contemplated the average of each subject on runs 101-105 and 196-200. The test utilized
in this statistical analysis was a nonparametric Wilcoxon signed-rank test, as the considered
data were mostly not normal distributed due to large between-subject differences.
3-7 Hypotheses
Based on a number of previous tracking experiments where the effects of both out-of-the-
window and motion cues were studied, together with earlier quasi-transfer-of-training studies,
the following hypotheses were formulated for this experiment:
H1: Training causes an improvement in performance and task proficiency in both experimen-
tal groups. Clear effects of training were expected to occur in both experimental groups during
the training phase, as seen in a number of previous training experiments (Refs. [7, 8, 16, 40]),
which are visible in improved performance (lower σ2e), increased control activity (higher σ2
u),
and higher crossover frequencies and phase margins. In the human operator modeling results,
it was expected to see adjustments in parameters that are known to be related to improved
performance (increased Ke, Kφ, lower human operator delays). Finally it was also expected
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
44 Experimental Methods
to see an increase on human operator linearity with training, measured with human operator’s
coherence.
H2: The presence of peripheral visual cues in training of control skills provides a feedback
channel of the controlled dynamics output. For the group trained with visual conditions
(Group V), previous studies (Refs. [1, 13, 14]) suggest that visual cues available in the
training phase provide a feedback channel for the roll angle and this was expected to be
visible in better performance (lower σ2e) and in the human operator parameters describing
the response to roll angle feedback. The roll gain Kφ and the roll delay τφ were expected to
be different from zero in the training phase of Group V.
H3: In the evaluation phase, the presence of motion allows reaching better task performance
levels. It is known that the addition of motion cues in a tracking task allows reaching better
levels of task performance [8, 9, 16]. This effect was expected to be mainly visible in perfor-
mance metrics with lower σ2e and higher σ2
u. In the human operator parameters, higher gains
(Ke and Kφ), lower delays (τe and τφ), and especially lower values of Tlead were expected as
a consequence of the lead information provided by the motion feedback.
H4: Adaption to motion conditions is faster for subjects who were trained with out-of-the-
window visual cues. It was expected that subjects in both groups would, at the end of the
evaluation phase, reach the same level of task proficiency, as a result of the extensive repetition
of the evaluation conditions with physical motion. However, this level of task proficiency was
expected to be reached earlier by subjects in Group V, meaning less hours of training would
be needed in a flight simulator with motion conditions. This would happen because transfer
of control skills to a motion condition was expected to be more effective for subjects who
trained with the presence of out-of-the-window visuals, as these cues would have created a
roll feedback response in training phase. This is supported by earlier findings that out-of-the-
window visuals function to a certain extent as a motion feedback channel [12]. This would
be visible in higher learning rates in the evaluation condition for subjects in Group V when
compared to subjects in Group NV and better performances in Group V immediately after
transfer.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
Chapter 4
Experimental Results
In this Chapter the results of the experiment are presented. In the Figures of this Chapter,
data from Group NV is shown in blue and data from Group V in red. In plots where
data evolution is shown over 200 runs, a black vertical line indicates the transfer after run
100. When learning curves are fitted to the data, which is done when Pearson’s correlation
coefficient is higher than 0.5, Pearson’s correlation coefficients for comparison of the curve
model and the real data are shown in the plot’s legend with the following organization ρ =
[ρtraining, ρevaluation]. Gray error bars are plotted indicating the 95% confidence intervals of
mean data for plots showing the evolution with the runs.
4-1 Tracking Performance
Tracking performance was measured with the variance of the roll error, i.e., the error presented
to the human operator on the PFD. The lower the value of σ2e , the better the performance.
Figure 4-1(a) shows the average variance of the tracking error per experiment run, together
with fitted learning curves and the 95% confidence intervals of the mean data. Furthermore,
to evaluate the performance improvement throughout the experiment, a decomposition in
components of tracking error variance was made, separating the contributions from the dis-
turbance forcing function, the target forcing function, and the remnant noise [39]. Results are
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
46 Experimental Results
shown, respectively, in Figs. 4-1(b), 4-1(c), and 4-1(d). The parameters of the fitted learning
curves in Fig. 4-1 are presented on the left side of Tables 4-1 to 4-4.
Average of group NV
Average of group V
Fit Group NV, ρ = [0.83, 0.93]
Fit Group V, ρ = [0.81, 0.92]
Evaluation RunsTraining Runs
σ2 e,deg
2
0 25 50 75 100 125 150 175 200
0.5
1
1.5
2
2.5
(a) Total tracking error variance.
Average of group NV
Average of group V
Fit Group NV, ρ = [0.74, 0.93]
Fit Group V, ρ = [0.83, 0.95]
Evaluation RunsTraining Runs
σ2 ed,deg
2
0 25 50 75 100 125 150 175 200
0.2
0.4
0.6
0.8
1
(b) Disturbance tracking error variance.
Average of group NV
Average of group V
Fit Group NV, ρ = [0.78, 0.84]
Fit Group V, ρ = [0.81, 0.75]
Evaluation RunsTraining Runs
σ2 et,deg
2
0 25 50 75 100 125 150 175 200
0
0.2
0.4
0.6
0.8
(c) Target tracking error variance.
Average of group NV
Average of group V
Fit Group NV, ρ = [0.81, 0.85]
Fit Group V, ρ = [0.68, 0.76]
Evaluation RunsTraining Runs
σ2 en,deg
2
0 25 50 75 100 125 150 175 200
0
0.5
1
1.5
(d) Remnant tracking error variance.
Figure 4-1: Measured tracking error variance and the disturbance, target, and remnant contri-butions.
Table 4-1: Learning curve parameters and statistical analysis for total tracking error.
σ2e
Learning Curve Parametersσ2e
Statistical SignificanceTraining Phase Evaluation Phase Training Transfer Evaluation
p0, deg2 pa, deg
2 F (×10−2) p0,deg2 pa, deg
2 F (×10−2) Ws Sig. Ws Sig. Ws Sig.Group NV 2.94 1.32 5.42 1.04 0.51 2.72 Group NV 55 ∗∗ 51 ∗ 55 ∗∗Group V 2.26 1.09 1.38 0.93 0.51 3.05 Group V 55 ∗∗ 55 ∗∗ 55 ∗∗
Table 4-2: Learning curve parameters and statistical analysis for disturbance tracking error.
σ2ed
Learning Curve Parametersσ2ed
Statistical SignificanceTraining Phase Evaluation Phase Training Transfer Evaluation
p0, deg2 pa, deg
2 F (×10−2) p0,deg2 pa, deg
2 F (×10−2) Ws Sig. Ws Sig. Ws Sig.Group NV 0.89 0.57 2.15 0.45 0.23 2.79 Group NV 55 ∗∗ 51 ∗∗ 54 ∗∗Group V 0.86 0.54 2.17 0.44 0.21 2.38 Group V 55 ∗∗ 55 ∗∗ 55 ∗∗
Table 4-3: Learning curve parameters and statistical analysis for target tracking error.
σ2et
Learning Curve Parametersσ2et
Statistical SignificanceTraining Phase Evaluation Phase Training Transfer Evaluation
p0, deg2 pa, deg
2 F (×10−2) p0,deg2 pa, deg
2 F (×10−2) Ws Sig. Ws Sig. Ws Sig.Group NV 0.66 0.33 4.94 0.28 0.15 3.12 Group NV 53 ∗∗ 49 ∗ 55 ∗∗Group V 0.66 0.31 6.49 0.26 0.16 4.63 Group V 54 ∗∗ 41 — 55 ∗∗
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
4-1 Tracking Performance 47
Table 4-4: Learning curve parameters and statistical analysis for remnant tracking error.
σ2en
Learning Curve Parametersσ2en
Statistical SignificanceTraining Phase Evaluation Phase Training Transfer Evaluation
p0, deg2 pa, deg
2 F (×10−2) p0,deg2 pa, deg
2 F (×10−2) Ws Sig. Ws Sig. Ws Sig.Group NV 1.13 0.31 7.35 0.26 0.08 1.41 Group NV 55 ∗∗ 30 — 54 ∗∗Group V 0.57 0.15 2.33 0.19 0.11 3.73 Group V 55 ∗∗ 45 — 47 ∗
Legend:
∗∗ indicates highly significant (p < 0.01) statistical differences between compared samples.
∗ indicates significant (0.01 ≤ p < 0.05) statistical differences between compared samples.
— indicates no significant (p ≥ 0.05) statistical differences between compared samples.
Some important conclusions can be drawn when looking at the tracking error results pre-
sented in Fig. 4-1. In the training phase, the performance level of Group V was always better
than the performance level reached by Group NV in every component of the tracking error,
suggesting that the presence of out-of-the-window visual cues indeed improved human opera-
tor performance, as expected from previous studies.[13, 12] In Fig. 4-1(a), it can be seen that
Group NV showed a steeper improvement in performance during the training phase, whereas
the learning curve for Group V was smoother, mainly because of the contribution of the rem-
nant error seen in Fig. 4-1(d). This is also seen in the learning rates in Table 4-1, as F in the
training phase was higher for Group NV. However, while Group NV stabilized their perfor-
mance approximately at σ2e = 1.32 deg2 around run 60, Group V kept a steady improvement
throughout the 100 training runs, reaching σ2e = 1.09 deg2 at the end of the training phase.
This suggests it took longer to adapt to the peripheral visual scene but the information it
provided allowed better performance, even in the first runs where a clear difference between
groups was already visible. Statistical data shown in Tables 4-1-4-4 for the training phase
confirm the significant improvements in performance. It is also important to notice that the
contribution of the disturbance forcing function was larger than the contribution of the target
forcing function, as seen in a comparison of Figs. 4-1(b) and (c), given that the experiment
had mainly a disturbance-rejection character.
Upon transfer, both groups showed a similar evolution in tracking performance, with an
instantaneous decrease in total error variance of 0.3 deg2 in Group NV and 0.2 deg2 in Group
V. This effect is also seen in the statistical analysis, reporting a significant difference in
compared samples of total error and disturbance components both groups. However, this was
not true in terms of target error for Group V and for remnant error for both groups, where
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
48 Experimental Results
no significant difference was found after transfer. This difference between groups suggests a
better adaptation to motion cues by Group V, given that no significant difference was found
in the target error component. This in itself suggests that transfer of control skills did happen
when subjects in Group V were transferred to the motion conditions.
For the evaluation phase, it can be seen that, as reported by numerous previous studies
[8, 9, 13], motion cues were more effective and they allowed for notable improvements in per-
formance levels of subjects in both groups. Both groups ended in the same level of asymptotic
task proficiency in terms of performance (total error of 0.51 deg2), showing a convergence of
control skills, a necessary premise to validate the results of any training experiment. The
confidence intervals for the evaluation phase were much smaller in amplitude and the Pear-
son’s coefficients of the learning curves were also higher in this phase, which suggest lower
subject variability as the experiment approached its end and subjects became more proficient
in the task. Group NV had a lower evaluation learning rate in the total error, target error,
and remnant error than Group V which means Group V was faster in learning how to use
motion cues. The improvement in performance for both groups during the evaluation phase
was validated by the statistical analysis. It can also be seen that, even though the target
error and the remnant error components decreased with motion, the main effect of motion
feedback was to decrease the disturbance tracking error. This was also observed in previous
tracking tasks [13, 31].
4-2 Control Activity
Figure 4-2 shows the evolution of the human operator control input throughout the 200 runs
performed, in a similar way as it was done for the tracking performance. Table 4-5 shows,
on the left side, the fitted learning curves parameters and on the right side the statistical
analysis results.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
4-2 Control Activity 49
Average of group NV
Average of group V
Fit Group NV, ρ = [—, 0.72]
Fit Group V, ρ = [—, 0.93]
Evaluation RunsTraining Runs
σ2 u,deg
2
0 25 50 75 100 125 150 175 200
2
4
6
8
10
12
Figure 4-2: Average control input variance.
Table 4-5: Learning curve parameters and statistical analysis for control input.
σ2u
Learning Curve Parametersσ2u
Statistical SignificanceTraining Phase Evaluation Phase Training Transfer Evaluation
p0, deg2 pa, deg
2 F (×10−2) p0, deg2 pa, deg
2 F (×10−2) Ws Sig. Ws Sig. Ws Sig.Group NV n/a n/a n/a 3.94 9.10 4.14 Group NV 23 — 0 ∗∗ 14 —Group V n/a n/a n/a 1.67 8.59 1.18 Group V 33 — 5 ∗ 0 ∗∗
Legend:
∗∗ indicates highly significant (p < 0.01) statistical differences between compared samples.
∗ indicates significant (0.01 ≤ p < 0.05) statistical differences between compared samples.
— indicates no significant (p ≥ 0.05) statistical differences between compared samples.
Looking at Fig. 4-2, it can be seen that the improvement in performance made by both
groups was reached with approximately constant control activity throughout the runs, as no
significant difference exists. Learning curves were not fitted in the training phase control data
given the low Pearson’s correlation coefficient. This behavior was not expected when looking
at earlier training studies.
After the transfer of conditions, Group NV immediately showed a great increase in control
power, whereas Group V showed a smaller increase. Looking at the transfer line in Table 4-5,
it is seen that both increases were statistically significant but the increase in Group NV was
more significant.
For the evaluation phase, the increase in control activity because of motion was expected.
[8, 13, 31] Group NV registered much stronger inputs than Group V, in spite of an unusually
high spread of values. Notwithstanding, in the last runs, a leveling in the power of the control
signal was visible, approaching the levels of Group V. This behavior was not portrayed by
the learning curve, which translated in a lower Pearson’s coefficient, but it was visible in the
statistical analysis in Table 4-5 where there was no statistical significant difference between
σ2u at the beginning and at the end of evaluation, due to the considerable spread in the data.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
50 Experimental Results
4-3 Human Operator Modeling Results
The development of skill-based control behavior during training, transfer and evaluation was
further analyzed by identifying the human operator model considered in Fig. 3-1 for every
run performed by every subject. An assessment of the quality of the identified model was
performed in three steps. First, single Bode diagram plots for the human response are shown
where the model fits are compared to independently estimated frequency-domain describing
functions. Then, the coherence of the human operator at the start and at the end of both
experimental phases is compared. Afterwards, the VAF of the obtained model is presented.
Finally the results of the parameter estimation are shown, followed by the evolution of the
crossover frequencies and phase margins.
4-3-1 Model Fits and Describing Functions
Figure 4-3 presents Bode plots containing the frequency response of both response functions
(error and roll response) of the human operator control model for one subject in Group V. The
two upper row of plots in Fig. 4-3 show the data concerning the initial and final training runs,
and the two bottom row of plots show data concerning the initial and final evaluation runs. In
each Bode plot, the frequency response as calculated with the identified model is plotted with
a continuous line, and the correspondent describing function estimates, calculated with the
Fourier coefficients of the time signals, are plotted with circle markers. These plots represent
the general obtained results and are similar to the data from the remaining participants.
Comparing the error and roll responses in the upper rows of plots in Fig. 4-3, a large variation
is observed in the describing functions, together with discrepancies between the model and
the respective describing function. This indicates reduced consistency and linearity of control
behavior, which was expected given the naivety of the participants in the earlier training
runs. Comparing the blue and red lines in Figs. 4-3(a) and 4-3(c), higher gains were seen
in both responses, the neuromuscular frequency increased, and the neuromuscular damping
ratio decreased, which is a typical evolution in the acquisition of manual control skills by
task-naive controllers [30, 40].
Looking at the bottom rows of Fig. 4-3, it is clear that the quality of the model increased when
runs 1 and 100 are compared with runs 101 and 200, as the circular markers corresponding
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
4-3 Human Operator Modeling Results 51
Run 1Run 100
ω, rad/s
|Hpe(jω)|,−
100 10110-1
100
101
(a) Training – Error Response, Magnitude
Run 1Run 100
ω, rad/s
∠H
pe(jω),
deg
100 101-450
-175
100
(b) Training – Error Response, Phase
Run 1Run 100
ω, rad/s
|Hpφ(jω)|,−
100 10110-2
100
(c) Training – Roll Response, Magnitude
Run 1Run 100
ω, rad/s
∠H
pφ(jω),
deg
100 101-400
-75
250
(d) Training – Roll Response, Phase
Run 101Run 200
ω, rad/s
|Hpe(jω)|,−
100 10110-1
100
101
(e) Evaluation – Error Response, Magnitude
Run 101Run 200
ω, rad/s
∠H
pe(jω),
deg
100 101-450
-175
100
(f) Evaluation – Error Response, Phase
Run 101Run 200
ω, rad/s
|Hpφ(jω)|,−
100 10110-2
100
(g) Evaluation – Roll Response, Magnitude
Run 101Run 200
ω, rad/s
∠H
pφ(jω),
deg
100 101-400
-75
250
(h) Evaluation – Roll Response, Phase
Figure 4-3: Frequency Response of Error and Roll channels in the human operator control model.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
52 Experimental Results
to the describing functions are closer to the corresponding model lines. This means the
model became better in describing the human control behavior, as expected, because in runs
101 and 200 motion cues were available, which is known to improve the quality of the roll
feedback. Furthermore, people behave more linearly with training. From run 101 to 200,
human operators increased their error and roll gains, reduced the lead time constant, and
lowered the neuromuscular damping ratio.
It should also be pointed out that the models represented in Fig. 4-3 were obtained with
a time-domain fitting method, whereas the describing functions obtained with Fourier coef-
ficients only describe the human response on the 20 frequency-pairs of the disturbance and
target forcing functions. Therefore, a model fitted perfectly to the Fourier coefficients shown
above does not necessarily mean a better model for the collected data. In fact, time-domain
identification procedures are consistent and efficient, holding estimates for the model param-
eters closer to the true set [20].
4-3-2 Coherence
Figure 4-4 shows the coherence of human operator control inputs, averaged for every subject
in Group V, with respect to disturbance and target forcing functions, Γd and Γt, respectively.
The upper row shows the evolution in training, with a comparison of run 1 and 100, and
the bottom row shows the evolution in the evaluation phase, with a comparison of run 101
and 200. High values for the coherence were found for every phase of the experiment with
increasingly higher coherence values as the experiment advances, meaning higher linearity
of human control behavior. This increase was especially high in the training phase. No
consistent differences for coherence results were seen between run 100, 101 and 200.
4-3-3 Variance Accounted For
The results for the average VAF in each group throughout the experimental runs are shown
in Fig. 4-5. Values for VAF in tracking experiments are usually around 80% and 90% when
human operator data is averaged between repeated measurements [20]. In a training experi-
ment, the evolution in the model throughout the runs is crucial to evaluate the acquisition and
development of control skills and averaging results would mask that effect. Therefore, a model
was fit to each individual run without any averaging, lowering its VAF due to higher remnant
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
4-3 Human Operator Modeling Results 53
Run 1Run 100
ω, rad/s
Γd,—
10-1 100 1010.85
0.9
0.95
1
(a) Disturbance coherence - training phase.
Run 1Run 100
ω, rad/s
Γt,—
10-1 100 1010.85
0.9
0.95
1
(b) Target coherence - training phase.
Run 101Run 200
ω, rad/s
Γd,—
10-1 100 1010.85
0.9
0.95
1
(c) Disturbance coherence - evaluation phase.
Run 101Run 200
ω, rad/s
Γt,—
10-1 100 1010.85
0.9
0.95
1
(d) Target coherence - evaluation phase.
Figure 4-4: Average coherence with respect to disturbance and target frequencies for the initialand final runs in training and evaluation phases.
noise. In this experiment, the majority of the models obtained held VAFs between 60% and
80%, which is in accordance with previous training studies [8]. The reduced consistency and
linearity seen when comparing initial and final runs of both experiment phases in Fig. 4-3 and
Fig. 4-4 translated to lower VAFs, especially in the earlier runs of each phase. Furthermore,
especially low values of VAF (around 40% and 50%) are seen throughout training phase.
Subject in group NV
Average of group NV
Subject in group V
Average of group V
Evaluation RunsTraining Runs
VAF,%
0 25 50 75 100 125 150 175 200
40
50
60
70
80
90
100
Figure 4-5: Average Variance Accounted For of the estimated model.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
54 Experimental Results
4-3-4 Human Operator Model Parameters
In the previous sections an assessment of the fitness quality of the human operator model
was performed. It was concluded that the quality of the data increases with the number
of runs performed, which is something inherent to any training experiment with task-naive
participants. In this section, the estimated parameters of the error response, the roll response
and the neuromuscular system are presented. Fitted curves were included to show learning
trends whenever the respective Pearson’s coefficient was significant, with the parameters for
all fits made shown in Table 4-6. The average parameter estimation results do not show a
learning trend in the training phase, and therefore learning curves are not shown for this
phase.
Average of group NV
Average of group V
Fit Group NV, ρ = [—, 0.90]
Fit Group V, ρ = [—, 0.92]
Evaluation RunsTraining Runs
Ke,—
0 25 50 75 100 125 150 175 200
0
0.5
1
1.5
(a) Error gain.
Average of group NV
Average of group V
Fit Group NV, ρ = [—, 0.91]
Fit Group V, ρ = [—, 0.93]
Evaluation RunsTraining Runs
TLead,s
0 25 50 75 100 125 150 175 200
0
0.5
1
1.5
2
(b) Error lead time constant.
Average of group NV
Average of group V
Evaluation RunsTraining Runs
τ e,s
0 25 50 75 100 125 150 175 200
0.1
0.2
0.3
0.4
0.5
0.6
(c) Error Delay.
Figure 4-6: Average estimated parameters defining the error response of the human operatormodel.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
4-3 Human Operator Modeling Results 55
Average of group NV
Average of group V
Evaluation RunsTraining Runs
ωnm,rad/s
0 25 50 75 100 125 150 175 200
5
10
15
20
25
(a) Neuromuscular frequency.
Average of group NV
Average of group V
Evaluation RunsTraining Runs
ζ nm,—
0 25 50 75 100 125 150 175 200
0
0.2
0.4
0.6
0.8
1
(b) Neuromuscular damping ratio.
Figure 4-7: Average estimated parameters defining the human operator neuromuscular system.
Average of group NV
Average of group V
Fit Group NV, ρ = [—, 0.95]
Fit Group V, ρ = [—, 0.97]
Evaluation RunsTraining Runs
Kφ,—
0 25 50 75 100 125 150 175 200
−0.2
0
0.2
0.4
0.6
0.8
1
(a) Roll Gain.
Average of group NV
Average of group V
Evaluation RunsTraining Runs
τ φ,s
0 25 50 75 100 125 150 175 200
0
0.5
1
1.5
(b) Roll Delay.
Average of group NV
Average of group V
Fit Group NV, ρ = [—, 0.92]
Fit Group V, ρ = [—, 0.97]
Evaluation RunsTraining Runs
σ2 uφ/σ2 u,%
0 25 50 75 100 125 150 175 200
0
20
40
60
80
100
(c) Roll channel variance fraction.
Figure 4-8: Average estimated parameters defining the roll response of the human operatormodel, and roll channel variance fraction.
Table 4-6: Parameters for the evaluation phase learning curves shown in Figs. 4-6 and 4-8.
Ke Tlead, (s) Kφ σ2uφ/σ2
u, %
p0, — pa, — F(
×10−2)
p0, s pa, s F(
×10−2)
p0, — pa, — F(
×10−2)
p0, % pa, % F(
×10−2)
Group NV 0.50 1.11 3.46 1.04 0.38 5.75 0.16 0.65 4.91 13.80 79.57 3.91Group V 0.34 1.37 0.94 1.10 0.41 3.04 0.12 0.70 1.91 10.49 111.02 1.04
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
56 Experimental Results
Considering the human operator error parameters in the training phase, shown in Fig. 4-6, it
is clear that both the gain, the lead time constant and the delay did not show any consistent
trend, remaining approximately constant throughout the 100 runs. This is consistent with the
results of Ref. [16], where a similar task with the same controlled dynamics was performed.
These evolutions are also consistent to what was found in control input metrics, in Fig. 4-2.
Similarities between both groups indicate that peripheral visual cues do not affect the response
in this channel. For the evaluation phase, the presence of motion induced a significant increase
in the error gain for both groups, whereas the lead time constant decreased, as expected given
the lead information motion provides [9, 10, 11]. For the evaluation phase, the learning curve
parameters are shown in Table 4-6, where it can be seen that Group V had lower learning
rates in Kφ and Tlead suggesting less transfer of skills for Group V.
With respect to the neuromuscular system parameters, shown in Fig. 4-7, no effects of learning
were observed in the neuromuscular frequency, which was higher for Group V in training and it
increased after transfer. This was an expected effect of experimental conditions with motion
and can be seen as the human arm getting stiffer in motion conditions, corresponding to
the contraction of the arm and hand muscle [31]. Neuromuscular damping ratio decreased
throughout the experiment, which was also expected and it is a sign of task proficiency because
with decreasing damping ratios phase lag is slightly lower in the frequencies where the human
operator is actively controlling (frequencies around the crossover frequency).
Finally, the roll response parameters, plotted in Fig. 4-8, show that the use of the roll feedback
channel was much smaller in training than in evaluation, suggesting that out-of-the-window
visual cues were not as effective in providing a roll feedback channel as motion. It was seen
though a positive evolution in roll gain during training, which was initially zero and in the end
of training was slightly greater than zero. In terms of roll response delay, this parameter does
not have any significance when the roll gain is zero, which explains the large variability in
early training runs. In the final training runs a decreasing trend was seen and its values seem
to converge to the ones in the evaluation phase. However, a great group variability is visible
which does not allow drawing significant conclusions on the evolution of this parameter.
In Figure 4-8(c), the contribution of the roll feedback channel in the total control input
is shown. A modest and approximately constant motion contribution was seen in training
phase, which means peripheral visual cues slightly contributed for the development of a roll
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
4-3 Human Operator Modeling Results 57
feedback channel. In the evaluation phase, there was a clear and increasing dominance of the
roll feedback, and the final values (around 80%) were higher than what is usually expected
for this type of tasks [8], suggesting that this unstable dynamics required an extensive use of
motion cues.
4-3-5 Crossover Frequencies and Phase Margins
Figure 4-9 shows the average crossover frequencies and phase margins in the disturbance and
target open loops for both groups, calculated considering the identified model.
Average of group NV
Average of group V
Evaluation RunsTraining Runs
ωc,d,rad/s
0 25 50 75 100 125 150 175 200
1
2
3
4
(a) Disturbance crossover frequency.
Average of group NV
Average of group V
Evaluation RunsTraining Runs
ωc,t,rad/s
0 25 50 75 100 125 150 175 200
1
2
3
4
(b) Target crossover frequency.
Average of group NV
Average of group V
Evaluation RunsTraining Runs
φm
,d,deg
0 25 50 75 100 125 150 175 200
20
40
60
80
100
(c) Disturbance phase margin.
Average of group NV
Average of group V
Evaluation RunsTraining Runs
φm
,t,deg
0 25 50 75 100 125 150 175 200
20
40
60
80
100
(d) Target phase margin.
Figure 4-9: Average disturbance and target crossover frequencies and phase margins.
For the disturbance crossover frequency (Fig. 4-9(a)), a reduced but visible increase was seen
in both groups throughout training, which was congruent with the observed improvement
in tracking performance. In the final runs of the training phase Group NV had a slightly
higher disturbance crossover frequency than Group V, explained by the also slight difference
in control activity in the final runs of the training phase. When transferred to the evaluation
conditions, motion increased disturbance crossover frequency in both groups, which was an
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
58 Experimental Results
effect found in previous tracking experiments [13, 31]. Disturbance crossover frequency is
higher for Group NV, following the control activity results. For the target crossover frequency
(Fig. 4-9(b)), the results suggest again a slight increase throughout the training runs for
both groups, with higher crossover frequencies for Group NV in the training phase, and no
significant difference was seen after transfer.
With respect to the disturbance phase margin (Fig. 4-9(c)), a slight increase was seen in
both groups throughout training phase, with Group V showing higher values of phase margin
in the end of training phase. In evaluation, motion was seen to decrease disturbance phase
margin. For the target phase margin (Fig. 4-9(d)), Group V had higher values in the end of
training phase, following the results of the disturbance phase margin, and in the evaluation
phase motion cues caused an increase in φmt of both groups. Regarding the training phase,
the differences existing between groups in both disturbance and target phase margins were
not significant to allow any conclusions. Looking at the evaluation phase, effects of motion in
disturbance and target phase margins were expected given previous tracking studies [13, 31].
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
Chapter 5
Discussion
The goal of the research project analyzed in this paper was to evaluate the effectiveness of
out-of-the-window visual cues for the development of manual control skills in the absence of
motion feedback by task-naive controllers. This was achieved by measuring tracking behavior
in a roll tracking task with a controlled element that required significant human operator lead
equalization in a quasi-transfer-of-training experiment performed in the SIMONA Research
Simulator at TU Delft. The twenty participants were divided in two experimental groups and
different cues were given according to the group and the experiment phase. Group NV was
trained with only a PFD showing the tracking error whereas Group V was trained with both
the PFD and an out-of-the-window view. Both groups were then transferred to the same
motion-base evaluation condition with pure roll motion feedback. Tracking performance,
control effort, crossover frequencies, phase margins, and human operator modeling results in
each run and for each subject were used to evaluate the evolution of human operator control
behavior from the initial fully naive task exposure to the final task proficiency.
Considering the results from previous training experiments, it was expected to see clear ef-
fects of skill development in the training phase in both groups (Hypothesis H1)[8, 40, 30].
The progress seen in the first 100 runs showed a positive evolution in terms of task perfor-
mance, with a decrease in tracking error variance due to a consistent reduction of disturbance-
rejection, target-tracking and remnant error variances. The decrease in remnant error variance
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
60 Discussion
means the initial task-naive participants increased their linearity, which is a clear training ef-
fect. This was further corroborated by increasing disturbance and target coherence. In terms
of control activity, no clear tendency existed during training, which was consistent with the
human operator model parameters whose average estimates were approximately constant for
the first 100 runs. This was not expected considering previous training experiments with
task-naive participants, but it was congruent with the findings for the training experiment
described in Ref. [16], where the controlled dynamics were the same as used here. Therefore,
and given these findings, training causes an improvement in task proficiency but not neces-
sarily in terms of human control dynamics, whose parameters remained constant throughout
training. However, these parameters describe the human control behavior progressively better
as the human operator linearity increases with the number of training runs.
Based on results from earlier studies investigating the effect of out-of-the-window visual cues
on tracking task performance and human control behavior, it was hypothesized that subjects
in Group V would develop during training a roll feedback channel similar as the one created
when motion cues are available (Hypothesis H2). Analyzing the results obtained in this
experiment, out-of-the-window visuals helped subjects performing the control task, as Group
V had a lower tracking error variance in the training phase. However, the average estimates
of the motion gain Kφ were close to zero throughout training, meaning no strong roll feedback
channel was used. Only a slight increase in roll gain and a decreasing tendency in roll delay
were found, converging to the values obtained in the beginning of the evaluation phase and
thus suggesting a slight adaptation to motion cues in a fixed-base condition augmented with
peripheral visual cues. This reduced effect of a roll feedback channel created with peripheral
visual cues is not entirely consistent with previous studies on the effect of out-of-the-window
visual cues. A reason for this to happen might be due to the fact that the roll stimulus
provided by the checkerboards was weaker than the yaw visual stimulus provided in Ref. [12].
Furthermore, in Ref. [12], participants were not task-naive but experienced subjects who
logically attain better performance easier. Another cause might be the different dynamics
controlled, as the unstable roll dynamics used here require a control strategy with a stronger
need for lead equalization.
When transferring to motion conditions in the evaluation phase, both groups were seen to
achieve better performance using stronger control activity (Hypothesis 3). A clear evolution
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
61
in the human model parameters was also seen, with higher error and roll gains and lower lead
time constants, as a consequence of the lead information motion feedback provides. Therefore,
motion significantly helped human operators performing this control tracking task, therefore
confirming Hypothesis 3. Great differences were, however, found when groups were compared,
with motion and visual gains being higher for Group NV, together with lower lead constants.
This is explained with the significant differences in control activity levels between the groups in
the evaluation phase, as Group NV adopted a significantly stronger control activity. Stronger
inputs mean the dynamics are being more excited and thus better perceived by the human
operator. Therefore, stronger control inputs increase the benefits from motion feedback. No
clear explanation for this difference in control strategy was found, but the control strategy of
Group NV was less consistent than the one of Group V. The data spread was notably larger
for Group NV and the average control variance decreased in the end of evaluation, showing
an unusual parabolic shape in this phase of the experiment.
As a consequence of the absence of a roll feedback channel with visual cues in the training
phase, the benefit of training with visual cues was also not verified when transferring to
motion (Hypothesis H4). On one side, supporting this hypothesis, lower tracking errors and
higher learning rates in tracking error data were indeed found for subjects in Group V when
compared to subjects in Group NV, but on the other side this tendency was not confirmed
by the human operator model parameters, where higher learning rates were in fact found for
Group NV in the evaluation phase.
Looking at the overall results, there seems to exist a between-group imbalance. As mentioned,
no explicit group assignment was made and this seems to have led to uneven groups in manual
control skills predisposition. Considering control activity, Group NV had, in evaluation phase,
a significant in-group variance as some subjects had an abnormally high control activity, mak-
ing Group NV’s average significantly higher than Group V’s average. This result propagated
to the model parameter estimations, biasing the group comparison. To avoid this effect, a
suggestion for future training experiments is to perform a brief inquiry to the participants to
evaluate their propensity on developing manual control skills, and then assign them evenly
in the experimental groups, avoiding confounding factors. Subjects who were professional
athletes, played computer games, or had proficiency in playing musical instruments showed
higher propensity for this control task, reaching better overall tracking performances with
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
62 Discussion
faster learning rates, while keeping higher and more stable concentration levels. Having made
an initial selection based on these characteristics could have had consequences on the final
group comparison. Another solution to this problem is increasing the number of subjects per-
forming the experiment, guaranteeing more samples in each group, which in this experiment
was not possible given the limited availability of the SRS. This same factor justified the ex-
perimental setup chosen, with 50 tracking runs performed by each subject in each day, which
might have been too intensive for the subjects and might have led to concentration issues
that directly affected the results. It is advisable to make shorter sessions, without however
reducing the amount of runs in each experiment phase, as 100 runs seem to be the necessary
amount of practice needed to achieve task proficiency in a specific condition.
Finally, looking at the evaluation phase of both groups, the fact that a great improvement
was achieved with respect to training suggests an ineffective training setup. The control
skills learned in a fixed-base environment showed limited direct transfer to the moving-base
condition, which had been described in previous experiments for the case where no visual
scene was provided [8, 40, 30]. The data collected in this experiment supports the conclusion
that manual control skills developed during training with a peripheral visual scene also do
not positively transfer to a motion condition. While peripheral visual cues are beneficial in
terms of performance and simulator realism, they seem to not effectively create the feedback
channel motion utilizes and therefore they do not replace motion as a cue in an initial phase
of simulator-based training.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
Chapter 6
Conclusions
A quasi-transfer-of-training experiment was performed to evaluate to which extent out-of-the-
window visual cues are effective as an initial setting of simulator-based training, under the
hypothesis that a peripheral visual scene eases the developing of manual control skills with
physical motion. This easiness would be created because peripheral visual cues were expected
to develop a feedback channel similar to the one motion is known to introduce in human
operator control strategy. Twenty task-naive participants divided in two groups performed
a compensatory roll tracking task starting from a total task naivety condition until full task
proficiency. The training settings differed on the presence of visual cues providing roll rate
information to the subjects of one group. Both groups were then transferred to a motion-base
condition where pure roll motion feedback was provided. All individual tracking runs were
analyzed for tracking performance and control activity, and multimodal quasi-linear human
operator models were used to describe human control behavior. A clear difference between
groups was seen in terms of performance during training, with the group where peripheral
visual cues were provided having lower tracking errors. This suggests a benefit of using
peripheral visual cues in training of manual control skills. However, when the parameters
describing human operator control behavior were analyzed, the roll response could barely be
found when peripheral visual cues were provided, as the average roll response gain estimate
was found to be close to zero throughout the training runs. In the evaluation phase, subjects
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
64 Conclusions
trained with peripheral visual cues converged faster to their final performance level, but again
this effect is not visible in human model parameters, which indicates that developing control
skills with a peripheral visual scene does not hold any strong benefit when transferring to
a motion condition. Physical motion is ultimately the most relevant cue in simulator-based
training of control skills.
6-1 Recommendations for Future Training Experiments
Having conducted this 7-month project in which every step of a training experiment was
carefully developed, the following recommendations for future training experiments are given
based on the personal experience and the results obtained in this project.
• Perform an inquiry to determine the natural propensity on developing of manual control
skills, and then assign the subjects evenly in the groups;
• Increase the number of subjects in each group, to guarantee more data samples on each
group and thus increasing group homogeneity;
• Adopting a different experimental setup using shorter sessions, without however reduc-
ing the number of runs in each training condition.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
Bibliography
[1] R.J.A.W. Hosman and J.C. van der Vaart. Effects of vestibular and visual motion
perception on task performance. Acta Psychologica, 48(13):271 – 287, 1981.
[2] Gavan Lintern, Stanley N. Roscoe, and Jonathan E. Sivier. Display Principles, Control
Dynamics, and Environmental Factors in Pilot Training and Transfer. Human Factors:
The Journal of the Human Factors and Ergonomics Society, 32(3):299–317, June 1990.
[3] John R. Anderson. Acquisition of Cognitive Skill. Psychological Review, 89(4):369–406,
1982.
[4] J. Rasmussen. Skills, rules, and knowledge; signals, signs, and symbols, and other dis-
tinctions in human performance models. IEEE Transactions on Systems, Man, and
Cybernetics, SMC-13(3):257–266, May 1983.
[5] Keith R. Lohse and Alice F. Healy. Exploring the contributions of declarative and
procedural information to training: A test of the procedural reinstatement principle.
Journal of Applied Research in Memory and Cognition, 1(2):65–72, June 2012.
[6] William H. Levison, Roy E. Lancraft, and Andrew M. Junker. Effects of Simulator
Delays on Performance and Learning in a Roll-Axis Tracking Task. In Proceedings of
the Fifteenth Annual Conference on Manual Control, Dayton (OH), pages 168–186, 1979.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
66 BIBLIOGRAPHY
[7] Edward A Martin. The influence of tactual seat-motion cues on training and performance
in a roll-axis compensatory tracking task setting. Technical report, DTIC Document,
2008.
[8] Daan M. Pool, Gertjan A. Harder, and Marinus M. van Paassen. Effects of Simulator
Motion Feedback on Training of Skill-Based Control Behavior. Journal of Guidance,
Control, and Dynamics, 39(4):889–902, 2016.
[9] Richard S Shirley and Laurence R Young. Motion cues in man-vehicle control effects of
roll-motion cues on human operator’s behavior in compensatory systems with disturbance
inputs. IEEE Transactions on Man-Machine Systems, 9(4):121–128, 1968.
[10] Ronald A. Hess. Model for Human Use of Motion Cues in Vehicular Control. Journal
of Guidance, Control, and Dynamics, 13(3):476–482, 1990.
[11] Peter R. Grant, Bonnie Yam, Ruud J. A. W. Hosman, and Jeffery Allyn Schroeder. Effect
of Simulator Motion on Pilot Behavior and Perception. Journal of Aircraft, 43(6):1914–
1924, November/December 2006.
[12] Henk P. M. Peterse, Daan M. Pool, Marinus M. van Paassen, and Max Mulder. In-
teractions of Outside Visual Cues and Motion Cueing Settings in Yaw Tracking. In
Proceedings of the AIAA Modeling and Simulation Technologies Conference, Washing-
ton, D.C., number AIAA-2016-3370, June 2016.
[13] Daan M. Pool, Max Mulder, Marinus M. van Paassen, and J. C. van der Vaart. Effects
of Peripheral Visual and Physical Motion Cues in Roll-Axis Tracking Tasks. Journal of
Guidance, Control, and Dynamics, 31(6):1608–1622, 2008.
[14] Maxim C. Vos, Daan M. Pool, Herman J. Damveld, Marinus M. van Paassen, and Max
Mulder. Identification of Multimodal Control Behavior in Pursuit Tracking Tasks. In
Proceedings of the 2014 IEEE International Conference on Systems, Man, and Cyber-
netics, San Diego (CA), pages 69–74, October 2014.
[15] Frank M. Nieuwenhuizen, Max Mulder, Marinus M. van Paassen, and H. H. Bulthoff. In-
fluences of Simulator Motion System Characteristics on Pilot Control Behavior. Journal
of Guidance, Control, and Dynamics, 2013.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
BIBLIOGRAPHY 67
[16] Peter M. T. Zaal, Alexandru Popovici, and Melinda A. Zavala. Effects of False Tilt Cues
on the Training of Manual Roll Control Skills. In Proceedings of the AIAA Modeling and
Simulation Technologies Conference, Kissimmee (FL), number AIAA-2015-0655, 2015.
[17] Anton van Lunteren. Identification of Human Operator Describing Function Models with
One or Two Inputs in Closed Loop Systems. PhD thesis, Delft University of Technology,
Faculty of Mechanical Engineering, January 1979.
[18] Marinus M. van Paassen. Biophysics in Aircraft Control, A Model of the Neuromuscu-
lar System of the Pilot’s Arm. PhD thesis, Delft University of Technology, Faculty of
Aerospace Engineering, 1994.
[19] Frank M. Nieuwenhuizen, Peter M. T. Zaal, Max Mulder, Marinus M. van Paassen, and
Jan A. Mulder. Modeling Human Multichannel Perception and Control Using Linear
Time-Invariant Models. Journal of Guidance, Control, and Dynamics, 31(4):999–1013,
July-August 2008.
[20] Peter M. T. Zaal, Daan M. Pool, Q. P. Chu, Marinus M. van Paassen, Max Mulder,
and Jan A. Mulder. Modeling Human Multimodal Perception and Control Using Ge-
netic Maximum Likelihood Estimation. Journal of Guidance, Control, and Dynamics,
32(4):1089–1099, 2009.
[21] Duane T McRuer, Dunstan Graham, ES Krendel, and William Reisener Jr. Human
pilot dynamics in compensatory systems. theory, models and experiments with controlled
element and forcing function variations. Wright-Patterson AFB (OH): Air Force Flight
Dynamics Laboratory, 1965.
[22] Valentin Dragoi and Chieyeko Tsuchitani. Electronic Textbook for Neurosciences, 2000.
The University of Texas Health Science Center at Houston (UTHealth). Consulted online
at http://neuroscience.uth.tmc.edu/s2/chapter14.html on 26 July, 2016.
[23] Peter M. T. Zaal, Frank M. Nieuwenhuizen, Marinus M. van Paassen, and Max Mulder.
Modeling Human Control of Self-Motion Direction With Optic Flow and Vestibular
Motion. IEEE Transactions on Cybernetics, 43(2):544–556, April 2013.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
68 BIBLIOGRAPHY
[24] Robert L. Stapleford, Duane T. McRuer, and R. E. Magdaleno. Pilot Describing Function
Measurements in a Multiloop Task. IEEE Transactions on Human Factors in Electronics,
8(2):113–125, June 1967.
[25] Johannes Cornelis Van der Vaart. Modelling of perception and action in compensatory
manual control tasks. TU Delft, Delft University of Technology, 1992.
[26] Ruud J. A. W. Hosman. Pilot’s Perception and Control of Aircraft Motions. PhD thesis,
Delft University of Technology, Faculty of Aerospace Engineering, 1996.
[27] Duane T. McRuer and Henry R. Jex. A Review of Quasi-Linear Pilot Models. IEEE
Transactions on Human Factors in Electronics, HFE-8(3):231–249, September 1967.
[28] William H. Levison and Andrew M. Junker. A Model for the Pilot’s use of Motion
Cues in Roll-Axis Tracking Tasks. BBN Report 3528, Bolt Beranek and Newman Inc.,
Cambridge (MA), April 1977.
[29] William H Levison. A model for the pilot’s use of roll-axis motion cues in steady-state
tracking tasks. Technical report, DTIC Document, 1978.
[30] Daan M. Pool, Gertjan A. Harder, Herman J. Damveld, Marinus M. van Paassen, and
Max Mulder. Evaluating Simulator-Based Training of Skill-Based Control Behavior using
Multimodal Operator Models. In Proceedings of the 2014 IEEE International Conference
on Systems, Man, and Cybernetics, San Diego (CA), pages 3868–3873, October 2014.
[31] Peter M. T. Zaal, Daan M. Pool, Jaap de Bruin, Max Mulder, and Marinus M. van
Paassen. Use of Pitch and Heave Motion Cues in a Pitch Control Task. Journal of
Guidance, Control, and Dynamics, 32(2):366–377, 2009.
[32] Herman J. Damveld, David A. Abbink, Mark Mulder, Max Mulder, Marinus M. van
Paassen, Frans C. T. van der Helm, and Ruud J. A. W. Hosman. Measuring the Con-
tribution of the Neuromuscular System During a Pitch Control Task. In Proceedings
of the AIAA Modeling and Simulation Technologies Conference, Chicago (IL), number
AIAA-2009-5824, August 2009.
[33] Herman J. Damveld, Gijs C. Beerens, Marinus M. van Paassen, and Max Mulder. Design
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
BIBLIOGRAPHY 69
of Forcing Functions for the Identification of Human Control Behavior. Journal of
Guidance, Control, and Dynamics, 33(4):1064–1081, July 2010.
[34] Edwin M Robertson, Alvaro Pascual-Leone, and Daniel Z Press. Awareness modifies the
skill-learning benefits of sleep. Current Biology, 14(3):208–212, 2004.
[35] Shailesh S. Kantak and Carolee J. Winstein. Learning-performance distinction and mem-
ory processes for motor skills: A focused review and perspective. Behavioural Brain
Research, 228(1):219–231, 2012.
[36] Walter R. Berkouwer, Olaf Stroosma, Marinus M. van Paassen, Max Mulder, and Jan A.
Mulder. Measuring the Performance of the SIMONA Research Simulator’s Motion Sys-
tem. In Proceedings of the AIAA Modeling and Simulation Technologies Conference and
Exhibit, San Francisco (CA), number AIAA 2005-6504, August 2005.
[37] Olaf Stroosma, Marinus M. van Paassen, Max Mulder, and Ferdinand N. Postema. Mea-
suring Time Delays in Simulator Displays. In Proceedings of the AIAA Modeling and
Simulation Technologies Conference and Exhibit, Hilton Head (SC), number AIAA-2007-
6562, 2007.
[38] Daan M. Pool, Peter M. T. Zaal, Marinus M. van Paassen, and Max Mulder. Effects of
Heave Washout Settings in Aircraft Pitch Disturbance Rejection. Journal of Guidance,
Control, and Dynamics, 33(1):29–41, 2010.
[39] Henry R. Jex, R. E. Magdaleno, and Andrew M. Junker. Roll Tracking Effects of G-
vector Tilt and Various Types of Motion Washout. In Proceedings of the Fourteenth
Annual Conference on Manual Control, pages 463–502, 1978.
[40] Joost C. F. de Winter, Dimitra Dodou, and Max Mulder. Training Effectiveness of Whole
Body Flight Simulator Motion: A Comprehensive Meta-Analysis. The International
Journal of Aviation Psychology, 22(2):164–183, 2012.
[41] Robert L Stapleford, Richard A Peters, and Fred R Alex. Experiments and a model for
pilot dynamics with visual and motion inputs. 1969.
[42] Marinus M. van Paassen and Max Mulder. Identification of Human Operator Control
Behaviour in Multiple-Loop Tracking Tasks. In Proceedings of the Seventh IFAC/IFIP/I-
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
70 BIBLIOGRAPHY
FORS/IEA Symposium on Analysis, Design and Evaluation of Man-Machine Systems,
Kyoto Japan, pages 515–520. Pergamon, 1998.
[43] Franciscus Albertus Martinus van der Steen. Self-motion perception. TU Delft, Delft
University of Technology, 1998.
[44] Thomas Brandt, Peter Bartenstein, Anika Janek, and Marianne Dieterich. Reciprocal
inhibitory visual-vestibular interaction. visual motion stimulation deactivates the parieto-
insular vestibular cortex. Brain, 121(9):1749–1758, 1998.
[45] Peter MT Zaal, Frank M Nieuwenhuizen, Marinus M van Paassen, and Max Mulder.
Modeling human control of self-motion direction with optic flow and vestibular motion.
IEEE transactions on cybernetics, 43(2):544–556, 2013.
[46] Daan M. Pool and Peter M. T. Zaal. Between-Subject Variability in Transfer-of-Training
of Skill-Based Manual Control Behavior. In Proceedings of the IEEE International Con-
ference on Systems, Man, and Cybernetics, Hong Kong, pages 1095–1099, October 2015.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
Appendix A
Processing Experimental Results
In this Chapter, multiple aspects of the experimental data processing are carefully explained,
to justify the decisions made, to detail the methods followed and to check the results obtained.
Firstly, in Section B-1, the results of the two participants who were excluded from the ex-
periment set are presented and analyzed to justify their removal from the final experimental
data.
Secondly, in Section B-2, the methods followed to solve issues appearing during identification
techniques are described.
Thirdly, in Section B-3, an overview of the individual runs that were excluded from the final
data set is made, with an explanation on the reasons for that exclusion.
This Chapter ends in Section B-4, alternative identification methods are described and the
results shown, to cross-check the results obtained and presented in the main thesis.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
72 Processing Experimental Results
A-1 Excluding Subjects
The initial experiment planning consisted on having ten subjects in each group, who would
perform the experiment in five weeks. However, in week four (the second week of group V),
two subjects registered problems in their performance in the experiment, problems that are
explained and detailed in this section. For this reason, the two initial test subjects were
included in the final data set, which changed the experiment planning for the last week in
order to reach the ten subjects in each experimental group in the final considered set.
These subjects were excluded from the final set of considered subjects for different reasons,
which are individually explained in the following subsections.
A-1-1 Excluded Subject 1
The main reason leading to the exclusion of this subject of group V was his inconsistencies in
performance and difficulties in keeping the concentration at a steady, constant level. In Fig.
A-1(a) the tracking error variance is shown for the 200 tracking runs performed, and in Fig.
A-1(b) the respective control input variance.
Excluded Subject 1
Fit, ρ = [0.32, 0.67]
Average Group V
Evaluation RunsTraining Runs
σ2 e,deg
2
0 25 50 75 100 125 150 175 2000
1
2
3
4
5
(a) Tracking error variance.
Excluded Subject 1
Fit, ρ = [0.27, 0.40]
Average Group V
Evaluation RunsTraining Runs
σ2 u,deg
2
0 25 50 75 100 125 150 175 2000
10
20
30
40
(b) Control input variance.
Figure A-1: Tracking error and control input variances for Excluded Subject 1.
Analyzing these results, it can be seen that this subject was highly inconsistent in terms
of performance throughout the first 150 runs, with his training learning curve having the
lowest training Pearson correlation coefficient in the entire experiment. In the training phase,
no learning is present and even worse performances were obtained in the middle of this
phase when compared to the beginning of the experiment. In the evaluation phase, a clear
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
A-1 Excluding Subjects 73
difference is seen between both experimental days: in the first day of the evaluation phase
(runs 101—150), the performance of this subject was approximately the same as in training
phase, i.e., motion did not introduce any improvements on the tracking performance, and this
was the only subject where such behavior was observed after transfer to motion. However,
another incongruity was the substantial improve (both in tracking performance results and
consistency) that happened in the last day, runs 150 to 200, with results following the group’s
average. Control input variance is consistence with the observations previously made.
If human operator modeling results are analyzed for this subject, the inconsistencies in perfor-
mance clearly affect the estimates made on his model. In Fig. A-2 the VAF of the identified
model for the 200 runs is shown. If compared with the other subjects in this group, excep-
tionally low values of VAF can be seen in the training phase, portraying the inconsistency
seen before in this subject control behavior. Even in evaluation phase the VAF of the model
is significantly below average compared to the entire group.
Excluded Subject 1
Average Group V
Evaluation RunsTraining Runs
VAF,%
0 25 50 75 100 125 150 175 20020
30
40
50
60
70
80
Figure A-2: VAF of the identified model for Excluded Subject 1.
Figure A-3 plot the estimates for the parameters that define the model found. Again, the
fact that no specific trend is found and highly inconsistent estimates were obtained justifies
the exclusion of this subject from the final considered data set of the experiment.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
74 Processing Experimental Results
Excluded Subject 1
Fit, ρ = [— , 0.38]
Average Group V
Evaluation RunsTraining Runs
Ke,—
0 25 50 75 100 125 150 175 2000
0.2
0.4
0.6
0.8
1
1.2
(a) Error gain.
Excluded Subject 1
Fit, ρ = [— , 0.63]
Average Group V
Evaluation RunsTraining Runs
TLead,s
0 25 50 75 100 125 150 175 200−1
0
1
2
3
4
(b) Error lead time constant.
Excluded Subject 1
Average Group V
Evaluation RunsTraining Runs
τ e,s
0 25 50 75 100 125 150 175 2000
0.1
0.2
0.3
0.4
0.5
(c) Error Delay.
Excluded Subject 1
Average Group V
Evaluation RunsTraining Runs
ωnm,rad/s
0 25 50 75 100 125 150 175 2005
10
15
20
25
30
(d) Neuromuscular frequency.
Excluded Subject 1
Average Group V
Evaluation RunsTraining Runs
ζ nm,—
0 25 50 75 100 125 150 175 2000
0.2
0.4
0.6
0.8
1
(e) Neuromuscular damping ratio.
Excluded Subject 1
Fit, ρ = [— , 0.01]
Average Group V
Evaluation RunsTraining Runs
Kφ,—
0 25 50 75 100 125 150 175 200−0.2
0
0.2
0.4
0.6
0.8
1
(f) Roll Gain.
Excluded Subject 1
Average Group V
Evaluation RunsTraining Runs
τ φ,s
0 25 50 75 100 125 150 175 2000
0.5
1
1.5
2
2.5
3
3.5
(g) Roll Delay.
Figure A-3: Estimated parameters of the model for Excluded Subject 1.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
A-1 Excluding Subjects 75
A-1-2 Excluded Subject 2
The main reason leading to the exclusion of this subject of group V was his bad performance
results throughout the experiment, which were worse than any other subject performing the
experiment, regardless of the group considered. In Fig. A-4(a) the tracking error variance is
shown for the 200 tracking runs performed, and in Fig. A-4(b) the respective control input
variance. In Fig. A-5 a detailed view of Fig. A-4(a) is presented, to better understand the
differences existing between this subject and the other subjects who were included in the
final data set of the experiment. In this figure, the average tracking error of group NV is also
shown.
Excluded Subject 2
Fit, ρ = [0.87, 0.73]
Average Group V
Evaluation RunsTraining Runs
σ2 e,deg
2
0 25 50 75 100 125 150 175 2000
10
20
30
40
(a) Tracking error variance.
Excluded Subject 2
Fit, ρ = [0.81, 0.23]
Average Group V
Evaluation RunsTraining Runs
σ2 u,deg
2
0 25 50 75 100 125 150 175 2000
5
10
15
(b) Control input variance.
Figure A-4: Tracking error and control input variances for Excluded Subject 2.
Excluded Subject 2
Average Group NV
Average Group V
Evaluation RunsTraining Runs
σ2 e,deg
2
0 25 50 75 100 125 150 175 2000
1
2
3
4
5
Figure A-5: Detail of the tracking error variance for Excluded Subject 2.
Firstly, in the first runs performed by this subject, the error variance was unusually high,
but that is understandable given that in the beginning the task is new and subjects are still
figuring out what should they do to control the dynamics. However, if the detailed plot
is analyzed, it can be seen that this subject tracking performance not only fails to show a
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
76 Processing Experimental Results
learning curve in training phase, but it also shows that this subject is way worse in terms
of tracking performance than the average of both groups. Including this subject in the final
data set would make the average data highly biased due to the influence of this subject in
group V, as this subject registered substantial differences in control behavior than the other
subjects who performed the experiment.
In Figure A-6 the VAF of the identified model for this subject is shown. Good results for the
VAF were obtained, meaning the control behavior of this subject was coherent, in opposition
to what was found for the previous excluded subject. The reason to exclude this subject
was not lack of consistency, but a lack of natural manual control skills that would made the
groups unbalanced if this subject would be included in the final data set.
Excluded Subject 2
Average Group V
Evaluation RunsTraining Runs
VAF,%
0 25 50 75 100 125 150 175 20040
50
60
70
80
90
Figure A-6: VAF of the identified model for Excluded Subject 2.
Figure A-7 plot the estimates for the parameters that define the model found. This subject bad
performance results are mainly visible in lower gains (error and roll) and in the neuromuscular
parameters.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
A-1 Excluding Subjects 77
Excluded Subject 2
Fit, ρ = [— , 0.69]
Average Group V
Evaluation RunsTraining Runs
Ke,—
0 25 50 75 100 125 150 175 2000
0.2
0.4
0.6
0.8
1
1.2
(a) Error gain.
Excluded Subject 2
Fit, ρ = [— , 0.65]
Average Group V
Evaluation RunsTraining Runs
TLead,s
0 25 50 75 100 125 150 175 200−1
0
1
2
3
4
5
(b) Error lead time constant.
Excluded Subject 2
Average Group V
Evaluation RunsTraining Runs
τ e,s
0 25 50 75 100 125 150 175 2000
0.2
0.4
0.6
0.8
1
(c) Error Delay.
Excluded Subject 2
Average Group V
Evaluation RunsTraining Runs
ωnm,rad/s
0 25 50 75 100 125 150 175 2000
5
10
15
20
25
(d) Neuromuscular frequency.
Excluded Subject 2
Average Group V
Evaluation RunsTraining Runs
ζ nm,—
0 25 50 75 100 125 150 175 2000
0.2
0.4
0.6
0.8
1
(e) Neuromuscular damping ratio.
Excluded Subject 2
Fit, ρ = [— , 0.70]
Average Group V
Evaluation RunsTraining Runs
Kφ,—
0 25 50 75 100 125 150 175 200−0.2
0
0.2
0.4
0.6
0.8
(f) Roll Gain.
Excluded Subject 2
Average Group V
Evaluation RunsTraining Runs
τ φ,s
0 25 50 75 100 125 150 175 2000
1
2
3
4
5
(g) Roll Delay.
Figure A-7: Estimated parameters of the model for Excluded Subject 2.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
78 Processing Experimental Results
A-2 Solving Identification Issues
As mentioned in the paper on Part I, the identification algorithm used to obtain the human
operator control behavior model was a time-domain Maximum Likelihood Estimation algo-
rithm, where the initial condition of a Gauss-Newton optimization procedure was provided by
a genetic algorithm. Given the gradient-based character of the Gauss-Newton algorithm used
to minimize the likelihood function, this method is not constrained to a feasibility domain
which means it can result in a solution without physical meaning, like estimating the natural
frequency of the neuromuscular system higher than the highest of the target and disturbance
frequencies, or holding neuromuscular damping ratios higher than 1 or lower than 0.
To avoid those meaningless solutions, the following strategy was adopted.
1. 10 repetitions of the genetic algorithm provide 10 initial conditions for the Gauss-
Newton optimization, which then yield 10 final parameter sets for the human operator
control behavior model;
2. The 10 final solutions are ordered in terms of the final value of the likelihood function,
and the solution with the lowest likelihood is taken as the best estimate of the model
parameters;
3. If the referred solution satisfies the physical restrictions inherent to the model, then its
estimates are taken as the final set of parameters for that tracking run;
4. If the referred solution fails to satisfy the physical restrictions inherent to the model
(namely, neuromuscular natural frequency higher than 30 rad/s and lower than 0 rad/s,
neuromuscular damping ratio higher than 1 or lower than 0), then the following solution
with lower likelihood is considered, and it is taken as the final solution if it satisfies the
physical restrictions. If not, the following solution is considered, and the process repeats
until one of the following conditions is true: either one of the Gauss-Newton solutions
satisfies the physical criterion, or all of them fail to meet the physical restrictions of the
model;
5. If the second situation is true, then the genetic algorithm estimation holding the lower
likelihood is considered as the final solution of the model parameters. Being a restricted-
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
A-2 Solving Identification Issues 79
domain optimization method, the solutions yielded by the genetic algorithm are by
definition within the domain defined in the algorithm setup;
6. If the solution with the lowest likelihood of the Gauss-Newton estimates is not considered
as the final solution, a comparison is made between the VAF of that solution and the
VAF of the considered solution (either another solution of the Gauss-Newton method
or the best genetic algorithm estimation). If a difference higher than 5% exists between
the VAF of both solutions, then a significant difference in the model parameters exists
between the considered models and the run is considered as an identification outlier.
In the set of 4000 runs analyzed, this situation only happened in three runs (group V,
subject 5, run 23; group V, subject 6, run 5; group V, Subject 9, run 89);
To better illustrate this process, in Fig. A-8 the Bode magnitude and phase plots of the error
channel response for a training run of Group NV are shown. In red, the frequency response
of the model that yielded the lowest likelihood in this run is shown. In blue, the frequency
response of the final considered model is plotted. And in black the Fourier coefficients of
this channel response are represented. It is clear that the model that best suits the Fourier
coefficients is the blue one, even if the Fourier coefficients do not fully describe the human
operator control behavior due to high levels of noise, as it is probably the case giving the
erratic distribution of the Fourier coefficient estimates.
Final ModelBest GN EstimationFC
ω, rad/s
|Hpe(jω)|,-
10-1 100 101 102 10310-2
10-1
100
101
(a) Error response - magnitude.
Final Model
Best GN Estimation
FC
ω, rad/s
∠(H
pe(jω)),deg
100 101
-400
-300
-200
-100
0
100
(b) Error response - phase.
Figure A-8: Bode plots of a run in group NV where the best Gauss-Newton estimated modelwas not taken as the final identified model.
In Table A-1, the characteristics and parameter estimates of both identified models are shown.
A negligible difference exists in the likelihood of both models, and the VAFs are also highly
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
80 Processing Experimental Results
similar. However, the differences in the parameter estimates are significant, especially in the
lead constant and in the neuromuscular parameters.
Table A-1: Comparison between both obtained models, regarding their characteristics and pa-rameter estimates
Likelihood, — VAF, % Ke, — Tlead, s τe, s ωnm, rad/s ζnm, —
Final Model -1902.26 71.07 0,14 1,83 0,36 7,02 0,98Best GN Estimate -1902.28 71.54 0,12 2,53 0,39 21,20 3,53
This example shows how the Gauss-Newton algorithm can converge into a non-realistic min-
imum of the likelihood function and justifies the need to replace such cases by another valid
estimate of the model.
In Fig. A-9 an overview of the 2000 runs performed by each group is presented in order to
indicate the origin of the final considered model. A black dot means the identified model
corresponds to the Gauss-Newton solution with the lowest likelihood. A blue dot means
another solution from the set of 10 Gauss-Newton estimates was considered. A red dot means
all the 10 Gauss-Newton estimates did not fulfill the physical restrictions and thus the best
genetic algorithm estimate was taken as the model of the human operator in that run. A
cross indicates the runs that were discarded as identification outliers, because the final model
was not feasible within the domain of the model parameters, and the other estimates held
a significant lower VAF when compared to the lowest likelihood solution. In Table A-2 the
same information is given in a numeric way, which can easily be converted in percentage given
that in each group and in each experiment phase, the total number of runs performed was
1000.
Table A-2: Overview of the origin of the final considered model for each group and phase of theexperiment.
Training EvaluationBest GN Other GN Best Genetic Excluded Best GN Other GN Best Genetic Excluded
Group NV 926 2 72 0 907 29 64 0Group V 815 66 116 3 880 56 64 0
From Fig. A-9 and Table A-2 it can be concluded that the lowest likelihood Gauss-Newton
solution was predominantly chosen as the final model. In the training phase of group V, the
genetic algorithm estimate was considered as the final model in a significant amount of times
(more than 10%), which is understandable given that in this phase physical motion cues are
not being provided to the human operator, while its control behavior model assumes a roll
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
A-2 Solving Identification Issues 81
feedback that is mainly given by motion.
It is also interesting to see the between-subject variability present in the results of the iden-
tification algorithms. Some subjects (namely subject 5 and 6 from group V, and subject 8
from group NV in the evaluation phase), clearly have more runs where the lowest likelihood
solution does not have physical meaning. This probably indicates that the control behavior of
these subjects does not exactly follow the considered structure, whether because they adopted
a different structure or because they are highly non-linear controllers.
Evaluation RunsTraining Runs
0 25 50 75 100 125 150 175 200
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
(a) Provenance of the identified model - Group NV.
Evaluation RunsTraining Runs
0 25 50 75 100 125 150 175 200
S1
S2
S3
S4
S5
S6
S7
S8
S9
S10
(b) Provenance of the identified model - Group V.
Figure A-9: Overview of the 200 runs performed by each subject showing on a color and symbolcode the provenance of the final identified model. Black dots mean the model of that runcorresponds to the Gauss-Newton estimate with the lowest likelihood, the blue dots mean themodel of that run corresponds to a lower likelihood Gauss-Newton estimate, and the red dotsmean the model was obtained considering the best genetic estimate. Crosses symbolize theexcluded runs.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
82 Processing Experimental Results
A-3 Variations on Identification Methods
The identification procedure described in the paper presented in Part I and further detailed in
Section E-2 was not the only method utilized to identify the human operator control behavior.
Different approaches were also made to check the validity of the estimates obtained with the
method described, which are presented in this section.
A-3-1 Identification on Multiple Consecutive Runs
As mentioned in previous sections of this dissertation, in tracking tasks as the one performed in
this experiment, it is common practice to perform the identification on average tracking runs
instead of individual tracking runs, in order to reduce the measurement noise. This option
was not considered in this experiment because an evolution throughout the 200 tracking
runs should be measured, justifying the need to model in each individual run the control
behavior of the human operator. However, to check what would be the results of performing
identification on multiple runs, an identification algorithm was developed to identify each of
the 5 consecutive runs performed by the subjects in Group V.
Time averaging of consecutive runs would be the easiest way of performing identification on
multiple runs. However, the use of different realizations of forcing functions in consecutive
runs invalidates this option. Therefore, another solution was needed, which consisted on
appending five consecutive runs in a single time trace, thus yielding time signals with 95 × 5
= 475 seconds, and considering only the last 81.92 seconds of data in each of those 5 runs for
the likelihood calculations, meaning the likelihood function considered 81.92 × 5 = 409.60
seconds of data. This is done to avoid taking in consideration in the likelihood function of
the transient response of the system. With this approach, 5 different consecutive runs are all
described by the same model, with the effect of measurement noise, non-linearities in control
behavior and control errors in the final model being minimized.
The results obtained for the VAF of the 5-run model in each of the individual runs is shown
in A-10, and the comparison between single-run and 5-runs estimates of the control behavior
model parameters of subjects in group V are shown in Figs. A-11 to A-13.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
A-3 Variations on Identification Methods 83
Individual Run
5 Runs Estimate
Evaluation RunsTraining Runs
VAF,%
Variance Accounted For
0 25 50 75 100 125 150 175 200
40
50
60
70
80
Figure A-10: VAF of the 5 runs model compared with the VAF obtained for individually identifiedruns, for Group V.
Individual Run
5 Runs Estimate
Evaluation RunsTraining Runs
Ke,—
Error Response - Gain
0 25 50 75 100 125 150 175 200
0
0.5
1
1.5
Evaluation RunsTraining Runs
TLead,s
Error Response - Lead Constant
0 25 50 75 100 125 150 175 200
0
0.5
1
1.5
2
Evaluation RunsTraining Runs
τ e,s
Error Response - Delay
0 25 50 75 100 125 150 175 200
0.1
0.2
0.3
0.4
0.5
0.6
Figure A-11: Single-run and 5-runs estimation of error response model parameters of Group V.
Evaluation RunsTraining Runs
ωnm,rad/s
Neuromuscular Frequency
0 25 50 75 100 125 150 175 200
5
10
15
20
25
Evaluation RunsTraining Runs
ζ nm,—
Neuromuscular Damping Ratio
0 25 50 75 100 125 150 175 200
0
0.2
0.4
0.6
0.8
1
Figure A-12: Single-run and 5-runs estimation of neuromuscular system parameters of Group V.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
84 Processing Experimental Results
Evaluation RunsTraining Runs
Kφ,—
Roll Response - Gain
0 25 50 75 100 125 150 175 200
-0.2
0
0.2
0.4
0.6
0.8
1
Evaluation RunsTraining Runs
τ φ,s
Roll Response - Delay
0 25 50 75 100 125 150 175 200
0
0.5
1
1.5
Figure A-13: Single-run and 5-runs estimation of roll response model parameters of Group V.
It can be seen that on a general view the 5 runs estimates do match the individually estimates,
which confirms the identification procedure followed.
Regarding the VAF obtained in both identification methods, the VAF obtained with the
individual runs is higher than when considering 5 runs. This makes sense because in the 5
runs model the weight of each run is averaged with the other 4 runs, meaning each run will
not be explained by the model as good as with an individual model. Higher values of VAF
considering multiple runs would only happen if a time average would be done, which was not
possible in this experiment as already mentioned.
Looking at the parameter results, the 5 runs estimates of the parameters approach an average
of the parameters themselves, which is consistent and validates the methods followed. Signif-
icant differences between both estimates are found in the error response delay in evaluation
phase and in roll response delay in training phase. In both cases, lower values were obtained
in the 5 runs model.
A-3-2 Single-Channel for Training of Group V
The next identification method followed was fitting a single-channel model in the training
phase of group V, where the participants had access to the peripheral visual cues but no motion
was provided. The results of this experiment suggest that the presence of the peripheral visual
cues do not significantly contribute to the roll feedback channel, therefore fitting a model with
only the error visual response would make sense to see the differences in the model parameters
and VAF, also to compare with the double-channel model estimate.
Figure A-14 shows the VAF of both models. It is visible that the double-channel model
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
A-3 Variations on Identification Methods 85
holds consistently better VAFs than the single-channel model, but that difference is small
and almost not significant. In the 1000 runs analyzed, in 87 runs the model of the single-loop
channel held a higher VAF than the double-loop channel. The average difference in VAF
between both models considering those 87 runs was 0.47 %. Considering the other 913 runs,
the double-loop channel model held, on average, a 0.68% higher VAF than the single-loop
channel model.
With Feedback Channel
Without Feedback Channel
Training Runs
VAF,%
Variance Accounted For
0 25 50 75 100
50
55
60
65
70
75
Figure A-14: Comparison of the VAF obtained with the single and double channel models fortraining of Group V.
In Figures A-15 and A-16 the error response and the neuromuscular system model parameters
are respectively shown.
With Feedback Channel
Without Feedback Channel
Training Runs
Ke,—
Error Response - Gain
0 25 50 75 100
0.2
0.25
0.3
0.35
0.4
Training Runs
TLead,s
Error Response - Lead Constant
0 25 50 75 100
0.8
1
1.2
1.4
1.6
1.8
2
Training Runs
τ e,s
Error Response - Delay
0 25 50 75 100
0.26
0.28
0.3
0.32
0.34
0.36
Figure A-15: Comparison between the error response model parameters in training of Group Vusing a single-channel or double-channel model structure.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
86 Processing Experimental Results
Training Runs
ωnm,rad/s
Neuromuscular Frequency
0 25 50 75 100
8
10
12
14
16
Training Runs
ζ nm,—
Neuromuscular Damping Ratio
0 25 50 75 100
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Figure A-16: Comparison between the neuromuscular system parameters in training of Group Vusing a single-channel or double-channel model structure.
While no clear difference is obtained for the neuromuscular parameters, which validates the
previous obtained results because these parameters should not depend on the number of
channels considered, some differences exist in the error gain and lead constant. The single-
channel model held slightly lower gains, which might be explained by the fact that in a double-
channel structure the effect of this gain is somewhat similar to the roll gain, and slightly higher
lead constants, meaning less lead information being generated by the peripheral visual cues.
These results are consistent with the overall experimental results. The importance of the roll
feedback channel created by the peripheral visual cues is reduced, but it exists. A single-
channel structure fails to better model the human operator control behavior.
A-3-3 Double-Channel for Training of Group NV
The final identification approach was to perform the opposite as done in the previous sub-
section, i.e., considering a double-channel for training of Group NV. During training of this
group no outside visual cue was provided and it is thus expected that a double-channel model
structure does not exist for this phase of group NV. Figure A-17 shows the VAF of the
double-channel model compared to the single-channel model VAF.
Higher VAFs are obtained for the double-channel structure, which is not entirely expected
given that there is no direct source of the roll feedback channel. In the 1000 runs analyzed,
in 61 the single-channel structure held higher VAFs than the double-channel, in an average
difference of 0.36 %. In the other 939 runs, the double-channel model held on average 0.75 %
higher VAFs than the single-channel model.
In Figures A-18 to A-20 the model parameters are shown, with a comparison of the single
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
A-3 Variations on Identification Methods 87
Without Feedback Channel
With Feedback Channel
Training Runs
VAF
Variance Accounted For
0 25 50 75 100
55
60
65
70
75
Figure A-17: Comparison of the VAF obtained with the single and double channel models fortraining of Group NV.
and double channel structures for the error response and neuromuscular system parameters,
and only the double-channel estimates for the roll response parameters.
It can be seen that the estimates of the error response and neuromuscular system param-
eters are highly similar, which is easily explained by the fact that the roll gain is almost
zero throughout the entire training phase. With the roll gain being zero, the double-channel
structure is reduced to the single-channel structure and therefore it makes sense that the
parameters are close to the previous estimates. These results are consistent with the exper-
iment conclusions: in this group, there is no roll feedback channel being created as the roll
gain is close to zero throughout the entire training phase. The fact that the double-channel
model holds higher VAFs for training phase of group NV than the single-channel model is
not consistent to the values found for the model parameters, and given the similarities in
the error response parameters, these differences can only be explained by the slight negative
values of Kφ which does not have a physical meaning.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
88 Processing Experimental Results
Without Feedback Channel
With Feedback Channel
Training Runs
Ke,—
Error Response - Gain
0 25 50 75 100
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Training Runs
TLead,s
Error Response - Lead Constant
0 25 50 75 100
0.5
1
1.5
2
Training Runs
τ e,s
Error Response - Delay
0 25 50 75 100
0.2
0.25
0.3
0.35
0.4
Figure A-18: Comparison between the error response model parameters in training of Group NVusing a single-channel or double-channel model structure.
Training Runs
ωnm,rad/s
Neuromuscular Frequency
0 25 50 75 100
5
10
15
Training Runs
ζ nm,—
Neuromuscular Damping Ratio
0 25 50 75 100
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Figure A-19: Comparison between the neuromuscular system parameters in training of GroupNV using a single-channel or double-channel model structure.
Training Runs
Kφ,—
Roll Response - Gain
0 25 50 75 100
-0.2
-0.1
0
0.1
0.2
Training Runs
τ φ,s
Roll Response - Delay
0 25 50 75 100
0
0.5
1
1.5
Figure A-20: Roll response model parameters for training of Group NV using a double-channelmodel structure.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
Appendix B
Experiment Documents
B-1 Call for Volunteers
1Challenge the future
Call for SIMONA volunteers!
Do you want to participate in a scientific research project
in the SIMONA Research Simulator?
From May 23 until June 24 we are running an experiment in
SIMONA to investigate how visual cues affect the training of
manual piloting skills in flight simulators.
For this study, we are looking for interested participants that
meet the following requirements:
• No prior piloting experience;
• Being right-handed;
• Being available for 4 2-hours sessions on 4 consecutive working
days of a single week in the experiment period.
I f you meet these requirements and want to participate in this
scientific endeavour, please send an e-mail to:
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
90 Experiment Documents
B-2 Consent Form
EXPERIMENT CONSENT F ORM
TRAINING EFFECTIVENESS OF FLIGHT SIMULATOR WITH OUTSIDE VISUAL CUES
Researcher: Miguel Mendes Supervisor: Daan Pool
Name: ___________________________ Age: ____
Have you participated in a tracking experiment before: Yes/No
Please provide your signature below to indicate that you agree to participate
in this experiment. Signing this form does not annul the responsibilities of the
researcher and Delft University of Technology towards you as a participant.
I hereby confirm that I have read the experiment briefing. Also, I affirm that I
understand the experiment instructions, and I declare that I voluntarily
participate in this experiment. Finally, I have been informed of the fact that I
can opt‐out of participating in the experiment at any time.
DATE: SIGNATURE:
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
B-3 Experiment Briefing 91
B-3 Experiment Briefing
EXPERIMENT BRIEFING
TRAINING EFFECTIVENESS OF FLIGHT SIMULATOR
WITH OUTSIDE VISUAL CUES
Thank you for your contribution to this scientific study! You will be participating in a tracking experiment in
the SIMONA Research Simulator, in which the effect of outside visual cues on the learning of control behavior
is investigated. This experiment is the central part of a Master of Science graduation project, and thus your
participation is not only highly appreciated, but it will also greatly contribute to a successful research outcome.
This briefing will introduce you to the experiment and explain in detail what is expected of you as a participant.
Please read it carefully and do not hesitate to ask for clarification if anything is not sufficiently clear.
GOAL OF THE EXPERIMENT
The goal of this experiment is to investigate how outside visual cues provided in a flight simulator can be used
to train pilots in the mastering of aircraft control techniques during early pilot training. Understanding this will
give insight on the value and necessity of motion systems in a flight simulator, especially in the early stages of
a pilot’s training.
ROLL TRACKING TASK
The task you will be trained in is a pure roll attitude tracking task. The aircraft will not move nor rotate in
other axis. Your goal to minimize a deviation of the current roll angle from a desired roll angle, as shown in
Figure 1.
FIGURE 1: SCHEMATIC REPRESENTATION OF A ROLL TRACKING TASK FIGURE 2: COMPENSATORY
DISPLAY
The error between the desired and current roll angle will be displayed on the primary flight display, shown in
Figure 2 and available on a screen located in front of your seat. The aircraft’s attitude is indicated by the fixed
wings symbol and the error is displayed by a moving line on a contrasting background, for an outside‐in display
configuration. It is your task to continuously try to keep the error as small as possible, using a sidestick located
on the right‐hand side of your seat. For giving roll control inputs, this sidestick should be moved from right to
left, where for an error as shown in Figure 2 an input to the left is required.
During the experiment, an outside scene may be projected in the aircraft windows. These images are
synchronized with the aircraft motion and help you minimizing the roll error.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
92 Experiment Documents
It is also possible that the simulator provides roll motion feedback during certain parts of the experiment,
obviously reproducing the aircraft motion. In such cases the researcher will inform you in advance, so that you
do not face anything unexpected. During the entire time you are in the simulator, you will always communicate
with the experiment supervisor using a microphone and (noise‐cancelling) headphones.
PROCEDURE
I� order to observe learning of the tracking task over a longer period of time, the experiment is divided over
four subsequent (working) days, with two‐hour experiment sessions on each day. In each session 50 tracking
runs of 100 seconds each are executed. In different sessions the supplied visual and motion cues may differ as
well. There will be a (coffee & cookies) break in the middle of each session (thus after the first 25 runs).
Though the chance is very small, this experiment may induce motion sickness in some individual. Therefore,
you will be asked to provide a score on the so‐called Misery Scale (MI��) after each set of 5 measurement runs.
The MI�� scale is provided in Table 1. Of course, if you wish to stop the experiment this is possible at any time.
Please do not hesitate to inform the experiment supervisor when you feel any signs of motion sickness.
TABLE 1: M�SC SCORES
Symptom Score
No problems 0
Slight discomfort but no specific symptoms 1
Dizziness, warm, headache, stomach
awareness, sweating, etc.
Vague 2
Some 3
Medium 4
Severe 5
Nausea Some 6
Medium 7
Severe 8
Retching 9
Vomiting 10
The subsequent procedure is followed in each tracking run:
1. The researcher applies the settings for the next run;
2. The researcher checks whether the participant is ready to proceed and initiates the run;
3. The participant performs the tracking task;
4. The researcher informs the participant of the performance in the previous run;
5. The researcher checks whether the participant can continue and, once every five runs, inquires after
signs of motion sickness and asks the participant to indicate a MI�� ��ore.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
B-4 Experiment Scheduling 93
B-4 Experiment Scheduling
Two weeks before the scheduled start of the experiment period, two test subjects performed
the experiment to check if it was properly designed and to detect eventual problems. The
procedures followed in this week were the same as the ones followed in the other weeks, to
guarantee that, if no problems were detected in the testing week, these two subjects could
still be included in the final considered set of subjects, if needed.
Initially, the five week experiment period allowed the participation of twenty subjects, which
would mean ten in each experimental group and therefore it would not be necessary to in-
clude the test subjects in the final data set of subjects. However, due to inconsistencies in
performance of two subjects in group V, an adjustment became imperative and therefore the
test subjects had to be included in the data set. To guarantee the same number of subjects
in both groups, in the final week three subjects performed the experiment in group V and
only one perform the experiment in group NV.
In the following pages, the experiment scheduling is shown for the testing week and the
five effective weeks of the experiment. Each subject performed eight sessions, two in each
day. Each session had 25 tracking runs and the two consecutive sessions made in each day
were separated by a 20 minute break, where the subjects would leave the simulator. The
participants could not leave the simulator during one session, but they could take small
breaks in between the tracking runs whenever they were feeling tired or if they would report
difficulties in keeping their concentration.
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
94 Experiment Documents
Table B-1: Experiment scheduling - testing week.
Week 0 09/05/2016 10/05/2016 11/05/2016 12/05/2016
Group NV Group NV Group V Group V9:00-9:50 Subject 1 Subject 1 Subject 1 Subject 1
Session 1 Session 3 Session 5 Session 7
9:50-10:10 Break Break Break Break
Group NV Group NV Group V Group V10:10-11:00 Subject 1 Subject 1 Subject 1 Subject 1
Session 2 Session 4 Session 6 Session 8
Group V Group V Group NV Group NV11:00-11:50 Subject 1 Subject 1 Subject 1 Subject 1
Session 1 Session 3 Session 5 Session 7
11:50-12:10 Break Break Break Break
Group V Group V Group NV Group NV12:10-13:00 Subject 1 Subject 1 Subject 1 Subject 1
Session 2 Session 4 Session 6 Session 8
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
B-4 Experiment Scheduling 95
Table B-2: Experiment scheduling - first week.
Week 1 23/05/2016 24/05/2016 25/05/2016 26/05/2016
Group NV Group NV Group NV Group NV9:00-9:50 Subject 2 Subject 2 Subject 3 Subject 3
Session 1 Session 3 Session 5 Session 7
9:50-10:10 Break Break Break Break
Group NV Group NV Group NV Group NV10:10-11:00 Subject 2 Subject 2 Subject 3 Subject 3
Session 2 Session 4 Session 6 Session 8
Group NV Group NV Group NV Group NV11:00-11:50 Subject 3 Subject 3 Subject 2 Subject 2
Session 1 Session 3 Session 5 Session 7
11:50-12:10 Break Break Break Break
Group NV Group NV Group NV Group NV12:10-13:00 Subject 3 Subject 3 Subject 2 Subject 2
Session 2 Session 4 Session 6 Session 8
13:00-14:00 Lunch Break Lunch Break Lunch Break Lunch Break
Group NV Group NV Group NV Group NV14:00-14:50 Subject 4 Subject 4 Subject 5 Subject 5
Session 1 Session 3 Session 5 Session 7
14:50-15:10 Break Break Break Break
Group NV Group NV Group NV Group NV15:10-16:00 Subject 4 Subject 4 Subject 5 Subject 5
Session 2 Session 4 Session 6 Session 8
Group NV Group NV Group NV Group NV16:00-16:50 Subject 5 Subject 5 Subject 4 Subject 4
Session 1 Session 3 Session 5 Session 7
16:50-17:10 Break Break Break Break
Group NV Group NV Group NV Group NV17:10-18:00 Subject 5 Subject 5 Subject 4 Subject 4
Session 2 Session 4 Session 6 Session 8
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
96 Experiment Documents
Table B-3: Experiment scheduling - second week.
Week 2 30/05/2016 31/05/2016 01/06/2016 02/06/2016
Group V Group V Group V Group V9:00-9:50 Subject 2 Subject 2 Subject 3 Subject 3
Session 1 Session 3 Session 5 Session 7
9:50-10:10 Break Break Break Break
Group V Group V Group V Group V10:10-11:00 Subject 2 Subject 2 Subject 3 Subject 3
Session 2 Session 4 Session 6 Session 8
Group V Group V Group V Group V11:00-11:50 Subject 3 Subject 3 Subject 2 Subject 2
Session 1 Session 3 Session 5 Session 7
11:50-12:10 Break Break Break Break
Group V Group V Group V Group V12:10-13:00 Subject 3 Subject 3 Subject 2 Subject 2
Session 2 Session 4 Session 6 Session 8
13:00-14:00 Lunch Break Lunch Break Lunch Break Lunch Break
Group V Group V Group V Group V14:00-14:50 Subject 4 Subject 4 Subject 5 Subject 5
Session 1 Session 3 Session 5 Session 7
14:50-15:10 Break Break Break Break
Group V Group V Group V Group V15:10-16:00 Subject 4 Subject 4 Subject 5 Subject 5
Session 2 Session 4 Session 6 Session 8
Group V Group V Group V Group V16:00-16:50 Subject 5 Subject 5 Subject 4 Subject 4
Session 1 Session 3 Session 5 Session 7
16:50-17:10 Break Break Break Break
Group V Group V Group V Group V17:10-18:00 Subject 5 Subject 5 Subject 4 Subject 4
Session 2 Session 4 Session 6 Session 8
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
B-4 Experiment Scheduling 97
Table B-4: Experiment scheduling - third week.
Week 3 06/06/2016 07/06/2016 08/06/2016 09/06/2016
Group NV Group NV Group NV Group NV9:00-9:50 Subject 6 Subject 6 Subject 7 Subject 7
Session 1 Session 3 Session 5 Session 7
9:50-10:10 Break Break Break Break
Group NV Group NV Group NV Group NV10:10-11:00 Subject 6 Subject 6 Subject 7 Subject 7
Session 2 Session 4 Session 6 Session 8
Group NV Group NV Group NV Group NV11:00-11:50 Subject 7 Subject 7 Subject 6 Subject 6
Session 1 Session 3 Session 5 Session 7
11:50-12:10 Break Break Break Break
Group NV Group NV Group NV Group NV12:10-13:00 Subject 7 Subject 7 Subject 6 Subject 6
Session 2 Session 4 Session 6 Session 8
13:00-14:00 Lunch Break Lunch Break Lunch Break Lunch Break
Group NV Group NV Group NV Group NV14:00-14:50 Subject 8 Subject 8 Subject 9 Subject 9
Session 1 Session 3 Session 5 Session 7
14:50-15:10 Break Break Break Break
Group NV Group NV Group NV Group NV15:10-16:00 Subject 8 Subject 8 Subject 9 Subject 9
Session 2 Session 4 Session 6 Session 8
Group NV Group NV Group NV Group NV16:00-16:50 Subject 9 Subject 9 Subject 8 Subject 8
Session 1 Session 3 Session 5 Session 7
16:50-17:10 Break Break Break Break
Group NV Group NV Group NV Group NV17:10-18:00 Subject 9 Subject 9 Subject 8 Subject 8
Session 2 Session 4 Session 6 Session 8
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
98 Experiment Documents
Table B-5: Experiment scheduling - forth week.
Week 4 13/06/2016 14/06/2016 15/06/2016 16/06/2016
Group V Group V Group V Group V9:00-9:50 Subject 6 Subject 6 Subject 7 Subject 7
Session 1 Session 3 Session 5 Session 7
9:50-10:10 Break Break Break Break
Group V Group V Group V Group V10:10-11:00 Subject 6 Subject 6 Subject 7 Subject 7
Session 2 Session 4 Session 6 Session 8
Group V Group V Group V Group V11:00-11:50 Subject 7 Subject 7 Subject 6 Subject 6
Session 1 Session 3 Session 5 Session 7
11:50-12:10 Break Break Break Break
Group V Group V Group V Group V12:10-13:00 Subject 7 Subject 7 Subject 6 Subject 6
Session 2 Session 4 Session 6 Session 8
13:00-14:00 Lunch Break Lunch Break Lunch Break Lunch Break
Group V Group V Group V Group V14:00-14:50 Subject 8 Subject 8 Subject 9 Subject 9
Session 1 Session 3 Session 5 Session 7
14:50-15:10 Break Break Break Break
Group V Group V Group V Group V15:10-16:00 Subject 8 Subject 8 Subject 9 Subject 9
Session 2 Session 4 Session 6 Session 8
Group V Group V Group V Group V16:00-16:50 Subject 9 Subject 9 Subject 8 Subject 8
Session 1 Session 3 Session 5 Session 7
16:50-17:10 Break Break Break Break
Group V Group V Group V Group V17:10-18:00 Subject 9 Subject 9 Subject 8 Subject 8
Session 2 Session 4 Session 6 Session 8
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues
B-4 Experiment Scheduling 99
Table B-6: Experiment scheduling - fifth week.
Week 5 20/06/2016 21/06/2016 22/06/2016 23/06/2016
Group NV Group NV Group V Group V9:00-9:50 Subject 10 Subject 10 Subject 10 Subject 10
Session 1 Session 3 Session 5 Session 7
9:50-10:10 Break Break Break Break
Group NV Group NV Group V Group V10:10-11:00 Subject 10 Subject 10 Subject 10 Subject 10
Session 2 Session 4 Session 6 Session 8
Group V Group V Group NV Group NV11:00-11:50 Subject 10 Subject 10 Subject 10 Subject 10
Session 1 Session 3 Session 5 Session 7
11:50-12:10 Break Break Break Break
Group V Group V Group NV Group NV12:10-13:00 Subject 10 Subject 10 Subject 10 Subject 10
Session 2 Session 4 Session 6 Session 8
13:00-14:00 Lunch Break Lunch Break Lunch Break Lunch Break
Group V Group V Group V Group V14:00-14:50 Subject 11 Subject 11 Subject 12 Subject 12
Session 1 Session 3 Session 5 Session 7
14:50-15:10 Break Break Break Break
Group V Group V Group V Group V15:10-16:00 Subject 11 Subject 11 Subject 12 Subject 12
Session 2 Session 4 Session 6 Session 8
Group V Group V Group V Group V16:00-16:50 Subject 12 Subject 12 Subject 11 Subject 11
Session 1 Session 3 Session 5 Session 7
16:50-17:10 Break Break Break Break
Group V Group V Group V Group V17:10-18:00 Subject 12 Subject 12 Subject 11 Subject 11
Session 2 Session 4 Session 6 Session 8
Training Effectiveness of Flight Simulators with Outside Visual Cues M.F.S. Mendes
100 Experiment Documents
B-5 Experiment Numbers
As a curiosity, some numbers about this transfer-of-training experiment are shown.
• 22 subjects performed the experiment over 24 days;
• A total of 176 hours were spent in the simulator, just for the final part of the experiment;
• 4400 tracking runs were made, meaning 380.000 seconds performing the control task;
• Each recorded variable had then 38.000.000 experimental points to be analyzed;
• The data recording system of the SRS created a file for each tracking run, meaning 4400
files were generated, with the digital size of such files being 8.504 kB;
• This means the total raw data occupies 35.6 GB;
• When a run data was conveniently processed and organized in a MatLab data structure,
each run had a corresponding workspace with the digital size of 1.091 kB;
• This means the processed data occupies 5.20 GB;
• The MLE identification method utilized to model the human operator control behavior
(10 genetic algorithm estimations and 10 Gauss-Newton optimizations) takes on average
15 minutes in each run;1
• This means the identification of the 4400 runs took something like 66.000 minutes to
be performed, which is 1.100 hours, almost 46 days of computational time;2
1It actually significantly depends on the experiment phase — Gauss-Newton method showed slower con-vergence times in training phase of group V, while the single-channel model identification of Group NV ontraining phase was significantly faster.
2Multiple computers were used in the identification process.
M.F.S. Mendes Training Effectiveness of Flight Simulators with Outside Visual Cues