Embed Size (px)
Citation preview
THE DEVELOPMENT AND EVALUATION OF A MODEL OF
TIME-OF-ARRIVAL UNCERTAINTY
by
BECKY L. HOOEY
A thesis submitted in conformity with the requirements
for the degree of Doctor of Philosophy
Graduate Department of Mechanical and Industrial Engineering
University of Toronto
© Copyright by Becky Hooey 2009
ii
The Development and Evaluation of a Model of Time-of-Arrival Uncertainty
Becky L. Hooey
Doctor of Philosophy, 2009 Department of Mechanical and Industrial Engineering, University of Toronto
Abstract
Uncertainty is inherent in complex socio-technical systems such as in aviation, military, and surface
transportation domains. An improved understanding of how operators comprehend this uncertainty is
critical to the development of operations and technology. Towards the development of a model of time of
arrival (TOA) uncertainty, Experiment 1 was conducted to determine how air traffic controllers estimate
TOA uncertainty and to identify sources of TOA uncertainty. The resulting model proposed that operators
first develop a library of speed and TOA profiles through experience. As they encounter subsequent
aircraft, they compare each vehicle!s speed profile to their personal library and apply the associated
estimate of TOA uncertainty.
To test this model, a normative model was adopted to compare inferences made by human observers to
the corresponding inferences that would be made by an optimal observer who had knowledge of the
underlying distribution. An experimental platform was developed and implemented in which subjects
observed vehicles with variable speeds and then estimated the mean and interval that captured 95% of
the speeds and TOAs.
Experiments 2 and 3 were then conducted and revealed that subjects overestimated TOA intervals for
fast stimuli and underestimated TOA intervals for slow stimuli, particularly when speed variability was
high. Subjects underestimated the amount of positive skew of the TOA distribution, particularly in
slow/high variability conditions. Experiment 3 also demonstrated that subjects overestimated TOA
uncertainty for short distances and underestimated TOA uncertainty for long distances. It was shown that
subjects applied a representative heuristic by selecting the trained speed profile that was most similar to
the observed vehicle!s profile, and applying the TOA uncertainty estimate of that trained profile.
iii
Multiple regression analyses revealed that the task of TOA uncertainty estimation contributed the most to
TOA uncertainty estimation error as compared to the tasks of building accurate speed models and
identifying the appropriate speed model to apply to a stimulus. Two systematic biases that account for
the observed TOA uncertainty estimation errors were revealed: Assumption of symmetry and aversion to
extremes. Operational implications in terms of safety and efficiency for the aviation domain are
discussed.
iv
Acknowledgements
I would like to express my gratitude to the members of my thesis committee (Professors Paul
Milgram, John Senders, and Greg Jamieson) for their guidance and insightful comments throughout
the entire PhD process. I am particularly grateful to Paul who was willing to support a long-distance
mentoring relationship as we attempted to close the chasm between engineering and psychology,
and between theoretical and applied research. I am honoured to have had the opportunity to share
many discussions with John Senders who brought interesting insights to the TOA uncertainty
problem. I am grateful to Greg for inspiring my PhD studies, as it was because of Greg!s seminar at
NASA Ames Research Center in 2003 that I pursued graduate studies at the University of Toronto. I
also extend my appreciation to my external committee members, Professors Ian Spence and Esa
Rantanen, for their time and effort.
I have been very fortunate to have the support of my colleague and friend, Dr. David Foyle of NASA
Ames Research Center, who supported this endeavour in every way possible including: endless
scientific discussions about time of arrival uncertainty, experimental methods, and data analyses;
facilitating the development and testing of the experiments; and his continual moral support and
encouragement. I would also like to thank Ron Miller for his programming expertise and support in
developing the experimental platform.
I am very grateful for the financial support that I received from the following sources: NSERC (Canada
Graduate Scholarship), Queen!s University (Marty Memorial Scholarship), University of Toronto
(Dissertation Completion Grant), and NASA Ames Research Center (support for the human-in-the-
loop experiments).
Finally, I would like to thank Brian Gore for sharing in all of the trials and tribulations along the way.
This could not have been completed without Brian!s support and encouragement.
v
The Development and Evaluation of a Model of Time-of-Arrival Uncertainty
Table of Contents
!"#$%&'$((((((((((((((((((((((((((((((((((((((((((((((((((((((( ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((())!
!'*+,-./01/2/+$# (((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((()3!
45!6789:;<::7=>8?@A?!99=B!C:D7@!E:FG4897!=G7H:=G:4@>6C8I::J@4=@?7845G=4!C:JHJ78>J(((((((((((((((;!
;(;:=G79@KF47=@G((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((;!
;(;(;:7)2/:,L:!%%)3&.:F+'/%$&)+$M:)+:$N/:!)%:7%&LL)':>&+&1/2/+$:JM#$/2 (((((((((((((((((((((((((((((((((((((((((((((; !
;(;(O::7)2/:,L:!%%)3&.:K/')#),+:JPQQ,%$:JM#$/2#((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((R!
;(;(R::K/')#),+:>&*)+1:&+0:J)$P&$),+:!-&%/+/## (((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((S!
;(;(S::K/L)+)+1:F+'/%$&)+$M(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((T!
;(O::98J8!945:@UV847=B8 ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((W!
45!6789:O<::8I689=>8G7:;:X:!G:=GB8J7=Y!7=@G:!GK:698C=>=G!9H:>@K8C:@A:7@!:
FG4897!=G7H:=G:!B=!7=@G:JF9A!48:D7!I=E:@689!7=@GJ ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((Z !
O(;::!B=!7=@G:JF9A!48:D7!I=E:@689!7=@GJ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((Z!
O(O:8I689=>8G7:[=75:!=9:79!AA=4:4@G79@C:D!74E:JFUV847:>!7789:8I6897J((((((((((((((((((((((((((( ;\!
O(O(;::>/$N,0(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( ;;!
"#"#$#$!%&'()*)+&,(-################################################################################################################################################################## $$!
"#"#$#"!!.++&'&(/- #################################################################################################################################################################### $$!
O(O(O::6%,'/0P%/ (((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( ;;!
"#"#"#$!!0,('12/*()1,################################################################################################################################################################ $$!
"#"#"#"!!3)-(&,*456+442!7&8)9'&()1,################################################################################################################################# $"!
"#"#"#:!!;<+4')=4,(&8!>')&8-################################################################################################################################################# $:!
"#"#"#?!!%1-(@-(/2A!349')4B ################################################################################################################################################### $?!
O(O(R::9/#P.$# (((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( ;S!
"#"#:#$!!>C.!74'(&),(A!D&(),E-############################################################################################################################################ $F!
"#"#:#"!!0,(4'G)4H-#################################################################################################################################################################### $I!
O(R:!:>@K8C:@A:7@!:FG4897!=G7H:8J7=>!7=@G(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( ;Z!
O(S::JF>>!9H((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( O;!
45!6789:R<::>875@KJ:!GK:>879=4J:7@:8B!CF!78::7@!:FG4897!=G7H:8J7=>!7=@G(((((((((((((((((((((((((((((((OO!
R(;::!UJ79!47=GY:K=J79=UF7=@G!C:69@6897=8J (((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( OO!
R(O::8I689=>8G7!C:>875@KJ:!GK:>879=4J(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( O]!
R(O(;::8^Q/%)2/+$&.:7&#*(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( OZ!
R(O(O::>/$%)'#:$,:83&.P&$/:F+'/%$&)+$M:4&.)"%&$),+ ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( R\!
45!6789:S<::8I689=>8G7:O:X:758:8AA847:@A:J688K:!GK:B!9=!U=C=7H:@G::7@!:FG4897!=G7H:
8J7=>!7=@G((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((R_!
S(;::J684=A=4:98J8!945:`F8J7=@GJ:!GK:5H6@758J8J (((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( R_!
S(O::8I689=>8G7!C:K8J=YG(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( RT!
S(R::>875@K ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( RW!
S(R(;::JP"a/'$#(((((((((((((((((((((((((((((((((((((((((((((((( ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( RW!
S(R(O::>&$/%)&.# (((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((( R]!
?#:#"#$!!.++&'&(/- #################################################################################################################################################################### :J!
?#:#"#"!!6()=/8) ########################################################################################################################################################################### :J!
vi
!"#"#$$%&'()*+&) """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" #,!
"#$#$#%!!&'()*+,-(.*'!/'+!0)/.'.'1#################################################################################################################################### $2!
"#$#$#$!!345).46!/'+!7*88*9:;<########################################################################################################################################### "%!
!"!$$-./012/ """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" !3!
!"!"4$$.567896)*$/:))* """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" !#!
!"!"3$$.567896)*$/:))*$;7<*'=5""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" !>!
!"!"#$$.567896)*$2?@"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" !A!
!"!"!$$.567896)*$2?@$;7<*'=5 """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" BC!
!"!"B$$.567896)*$2?@$;7<*'=$/D88)6&D""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" B3!
!"B$$EF/G0//F?H """""""""""""""""""""""""""""""""""""""""""""""" """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" BB!
!"B"4$$/+889&D$'I$-)5+J65"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" BB!
!"B"3$$17876967'<5$'I$6K)$G+&&)<6$.L:)&78)<6$9<*$/+MM)567'<5$I'&$N+6+&)$/6+*7)5"""""""""""""""""""""""""" B>!
!"B"#$$F8:J7(967'<5$I'&$6K)$H)L6$O)<)&967'<$@P7967'<$/D56)8"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" BA!
GQ@%2.-$BR$$.S%.-FT.H2$#$U$2?@$0HG.-2@FH2V$./2FT@2F?H$@HE$.S2-@%?1@2F?H"""""""""""""""""""""""""""""""B,!
B"4$$FH2-?E0G2F?H"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" B,!
B"4"4$$@**&)557<M$17876967'<5$'I$.L:)&78)<6$3 """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" B,!
B"3$$-./.@-GQ$W0./2F?H/ """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" >C!
B"#$$T.2Q?E """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" >4!
B"#"4$$/+XY)(65"""""""""""""""""""""""""""""""""""""""""""""""" """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" >4!
B"#"3$$.L:)&78)<69J$E)57M<"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" >4!
=#$#>#%!?@/A4!$/B!!34C48*<.'1!/!D.5)/)E!*6!F<44+!?)*6.84A###################################################################################### G%!
=#$#>#>#!!?@/A4!$5B!!HA(.I/(.'1!0JK!;'-4)(/.'(E!6*)!0)/.'4+!?)*6.84A############################################################### G"!
=#$#>#$#!!?@/A4!$-B!!HA(.I/(.'1!0JK!;'-4)(/.'(E!6*)!L49M!;'()/.'4+!?)*6.84A ############################################### GN!
B"!$$-./012/ """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" >,!
B"!"4$$%K95)$#9R$E)P)J':7<M$9$17X&9&D$'I$/:))*$%&'I7J)5"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" >,!
=#"#%#%!!HA(.I/(4+!O4/'!F<44+########################################################################################################################################### G2!
=#"#%#>!HA(.I/(4+!F<44+!P.'+*9A#################################################################################################################################### NQ!
B"!"3$$%K95)$#XR$.5678967<M$2?@$+<()&697<6D$I'&$6&97<)*$:&'I7J)5 """"""""""""""""""""""""""""""""""""""""""""""""""""""""" A3 !
=#"#>#%#!!HA(.I/(4+!0JK!;'-4)(/.'(E ################################################################################################################################ N>!
=#"#>#>#!!0JK!;'-4)(/.'(E!HA(.I/(.*'!H))*) ################################################################################################################## NG!
=#"#>#$!!HA(.I/(4+!0JK!&'(4)C/8!FEII4()E################################################################################################################### NN!
B"!"#$$%K95)$#(R$.5678967<M$2?@$0<()&697<6D$I'&$0<6&97<)*$%&'I7J)5"""""""""""""""""""""""""""""""""""""""""""""""""""" A, !
=#"#$#%!!&+4'(.6.-/(.*'!*6!0)/.'4+!/'+!;'()/.'4+!?)*6.84A ####################################################################################### RQ!
=#"#$#>!!S*))4-(!&+4'(.6.-/(.*'!*6!F<44+!?)*6.84############################################################################################################ RQ!
=#"#$#$!!F.I.8/).(E!T/(.'1A ##################################################################################################################################################### R%!
=#"#$#"!!HA(.I/(4+!0JK!P.'+*9######################################################################################################################################## R>!
B"B$$G?HG10/F?H$@HE$?%.-@2F?H@1$FT%1FG@2F?H/ """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" ZA!
GQ@%2.-$>R$$T?E.1$-.NFH.T.H2 """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""Z,!
>"4$2FT.$?N$@--F[@1$0HG.-2@FH2V$T?E.1$-./[email protected]$@HE$-.NFH.E """""""""""""""""""""""""""""""""""""""""""""""" Z,!
>"3$$/?0-G./$?N$2FT.$?N$@--F[@1$0HG.-2@FH2V$./2FT@2F?H$.--?-""""""""""""""""""""""""""""""""""""""""""""""" ,C!
>"#$$G@0/./$?N$2FT.$?N$@--F[@1$0HG.-2@FH2V$./2FT@2F?H$.--?-"""""""""""""""""""""""""""""""""""""""""""""""""" ,4!
>"#"4$$@55+8:67'<$'I$/D88)6&D """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" ,4!
>"#"3$$@P)&57'<$6'$.L6&)8)5""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" ,!!
GQ@%2.-$AR$$/0TT@-V\$1FTF2@2F?H/$@HE$FT%1FG@2F?H/""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""",>!
A"4$$T?E.1$@//0T%2F?H/$@HE$1FTF2@2F?H/ """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" ,A!
A"4"4$$/:))*$75$H'&89JJD$E756&7X+6)*"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" ,A!
A"4"3$$@P97J9X7J76D$'I$@((+&96)$/:))*$-)9*'+65 """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" ,Z!
vii
!"#"$%&'()%*+%,--'./0%123)-4/'245%674'(/4'*2%,89)-)7%4*%49)%:-*;*7)8%&<*=74);%:-*3)77%>*8)0
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" ??!
!"@%%A:6B,&CAD,E%C>:ECF,&CADG%HAB%,IC,&CAD""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""#JJ!
!"@"#"%%C(;0'3/4'*27%+*-%KL(/2%6--*- """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""#JJ!
!"@"@%%C(;0'3/4'*27%+*-%KL(/2=F)24-)8%,L4*(/4'*2%4*%GL;;*-4%,'-%&-/++'3%F*24-*0 """""""""""""""""""""#J@!
!"$%%H1&1B6%MCB6F&CADG """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""#JN!
FK,:&6B%OP%%B6H6B6DF6G """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""#J!!
,::6DMCF6G""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""##$!
,::6DMCQ%,#P%%6Q:6BC>6D&%#%:BA&AFAE """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""##$!
,::6DMCQ%R@P%%6Q:6BC>6D&%@%M6>ASB,:KCF%T16G&CADD,CB6""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""##O!
,::6DMCQ%R$P%%6Q:6BC>6D&%@%M6RBC6H%T16G&CADG"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""##?!
,::6DMCQ%RUP%>6&KAM%HAB%M6&6B>CDCDS%,F&1,E%G:66M%,DM%&A,%GFAB6G"""""""""""""""""""""""""""""""""#@J!
,::6DMCQ%F#P%%6Q:6BC>6D&%$%:BA&AFAE """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""#@#!
,::6DMCQ%F@P%%6Q:6BC>6D&%$%M6>ASB,:KCF%T16G&CADD,CB6""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""#@O!
,::6DMCQ%F$P%%6Q:6BC>6D&%$%M6RBC6H%T16G&CADG """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""#@?!
1
CHAPTER 1: TIME-OF-ARRIVAL (TOA) UNCERTAINTY IN COMPLEX
SOCIO-TECHNICAL SYSTEMS
1.1 INTRODUCTION
The need to predict time of arrival (TOA) is becoming increasingly prevalent in complex
socio-technical systems. For example, pilots and air traffic controllers predict TOA in order
to identify and prevent potential collisions, taxi dispatchers predict driver arrival times, and
weather forecasters use complex meteorological and oceanographic models to predict the
time at which a hurricane is expected to make landfall. These diverse domains all share the
common problem of TOA prediction uncertainty. In these cases, even when the data used
to make TOA predictions are essentially accurate at the specific point in time that they are
collected, there is some degree of uncertainty inherent in the prediction, since the world is
dynamic and continues to change. That is, even if the current location, course, and speed
are known, an object or event can change course and speed in the future, and this
introduces uncertainty into the predictions.
1.1.1 Time of Arrival Uncertainty in the Air Traffic Management System
The aviation domain has been selected for investigation in the current research in order to
ground it in a real-world problem. In the Air Traffic Management (ATM) system it is
necessary for air traffic controllers (ATC) to have an accurate view of possible future events
to prevent dangerous or undesirable situations and to optimize traffic flow. ATM is greatly
vulnerable to uncertainty, and controllers and pilots are frequently required to make
important decisions based on uncertain or incomplete information, especially when they
involve predictions of future system states. The sources of TOA uncertainty include factors
such as wind shifts, weather, route complexity, turbulence, pilots! intentions, pilot variability,
and prediction time span. Erroneous interpretation of this uncertain information carries with
it a potential for serious consequences.
Many attempts have been made to minimize uncertainty in the aviation system, such as
reliance on procedures and standardized routes intended to minimize variability and
increase predictability of aircraft movements. Also, great efforts have been taken to improve
the reliability of automation algorithms to minimize uncertainty (Kuchar, 2001; van Doorn,
2
Bakker & Meckiff, 2001). However, it is not possible to eliminate uncertainty from human-
automation systems altogether, particularly because of the latent variability in human and
environmental factors. Furthermore, this approach may not be sustainable, as new ATM
concepts that require greater precision and greater accuracy, such as reduced separation
minima and coordinated runway crossings, are introduced. These concepts require pilots
and controllers to anticipate trajectories of traffic to facilitate detection and avoidance of
potential conflicts and to make more efficient use of airspace and runways. However, all
these tasks inherently involve uncertainty, as future states within such complex system
cannot be known in advance.
Much research has been conducted to explore planning tasks that require anticipation and
prediction in uncertain, dynamic environments. Boudes and Cellier (2000) studied human
anticipation when controlling dynamic environments (such as ATC or power plants) in which
the operator does not have complete control over the evolution of the system. As per their
definition, !anticipating" involves evaluating the future state of a process and deciding,
according to the subsequent representations developed, which action is to be undertaken
and when. It is, of course, necessary to have an accurate view of possible future events to
prevent dangerous or undesirable situations and to better prepare for favourable conditions
for traffic evolution and optimization.
However, prediction and anticipation are not functions that people perform with high
precision, given known perceptual and cognitive limitations. For example, human operators
are limited in their ability to integrate information over time. Such tasks normally require
mental computation to extrapolate the future from the present and past, and may require that
future estimates be stored in working memory (Wickens & Hollands, 2000). Due to limits on
memory, people tend to give undue weight to early cues (primacy), those that support the
initial hypothesis (anchoring), and very recent cues (recency) (Kahneman, Slovic & Tversky,
1982; Wickens & Hollands, 2000). Furthermore, humans are not good estimators of
variability and are limited in their ability to perceive trends emerging in data that require
extrapolation of exponential or accelerating growth trends.
Non-specialized subjects (Schiff & Oldak, 1990) and trained experts, such as air traffic
controllers (Boudes & Cellier, 2000), also tend to exhibit a conservative bias in that they
make decisions that are risk averse. For example, the conservative bias is demonstrated by
3
an air traffic controller who is unaware of the exact location or speed of an aircraft and
leaves very large !space envelopes" around each aircraft. In essence, this decision bias is a
way of preparing optimal solutions to absorb unforeseen events, because it allows for a time
margin for solving problems. With large space envelopes, the controller has more time to
issue commands to the aircraft to resolve and avoid potential conflicts. However, it is
important to note that, although functional, the bias may also produce temporal errors and
introduce risk into the ATM system. For example, the temporal errors caused by the
conservative bias may lead controllers to ignore potential conflicts because he/she may
have a false sense that the aircraft are safely separated despite his/her lack of actual
knowledge of the aircraft"s precise location or speeds. The controller"s !safer" space
envelopes may also increase system risk in some cases, for example when the pilot must
carry out a go-around manoeuvre rather than land the aircraft to maintain the controller"s
conservative space envelope. Furthermore, the conservative decision bias may also limit
system efficiency and throughput by creating the need for holding patterns and departure
queues, both of which may be particularly undesirable as we face increasingly congested
airspace and runways.
1.1.2 Time of Arrival Decision Support Systems
To support TOA prediction in dynamic, uncertain environments, Decision Support Systems
(DSSs) are being developed and introduced into many complex domains, in an attempt to
reduce operator workload, improve decision-making, and reduce errors. For example,
automated conflict detection systems have been developed using predictor algorithms and
dynamic models to propagate aircraft state forward in time and predict conflicts over time
and space (Kuchar, 2001; Wickens & Morphew, 1997; Johnson, Battiste & Holland, 1999).
This requires some assumptions regarding the future intentions of each aircraft, such as
assuming that each aircraft will continue to fly at its current heading, speed, and altitude.
When such assumptions are violated, however, the inaccuracies may result in unpredicted
conflicts or unnecessary manoeuvres. In reality, in other words, such predictive conflict
detection systems are rarely 100% reliable. A recent estimate suggested that 85% reliability
is typical of a conflict detection system that predicts conflicts with long look-ahead times (i.e.,
20 minutes) in airspace subject to uncertainties arising from turbulence and future pilot
control actions between the time of alert and the predicted conflict (Xu, Wickens &
Rantanen, 2004).
4
Much of the research on automation reliability (see Wickens, 2000) has assumed a cueing
system that notifies the operator of impending events. In these systems, automation is often
considered to be either correct or incorrect – that is, the automation correctly cues the
human operator to an object that exists in the world, or does not. However, a DSS that
provides TOA predictions is more likely to have varying degrees of error, rather than error
versus no error. Yet, despite the known lack of perfect reliability and the inherent uncertainty
due to factors that make perfect prediction impossible (i.e., wind shifts, turbulence, pilot
variability, prediction time span, etc.), information about uncertainty associated with
prediction errors is not normally presented to the human decision maker (For one exception,
however, see Lee & Milgram, 2008).
1.1.3 Decision Making and Situation Awareness
In ill-structured, uncertain, and dynamic environments, automation has been less-than-
successful, and has even created serious problems, as the automation becomes "brittle"
and cannot adapt to unknown or unforeseen situations (Cohen, 1993). It fact, the very
features of real-world environments, such as their ill-structuredness, uncertainty, shifting
goals, dynamic evolution, time stress, multiple players, etc., typically defeat the static,
bounded models typically used by decision support systems (Cohen, 1993). Cohen
suggests that each decision involves a unique and complex combination of factors, which
seldom fits easily into a standard decision analytic template, a previously collected body of
expert knowledge, or a predefined set of linear constraints, and that users are sometimes
better than the aids in their ability to recognize and adapt to such complex patterns.
Hansman and Davison (2000) suggest that when decisions must be made based on
incomplete data [or uncertain information], humans are better at decision making than is
automation that follows rule-based strategies, because human operators can formulate a
more comprehensive picture and detect patterns indicating that a system is straying outside
normal limits. It is proposed that, even though humans are better at managing their tasks in
ill-structured, uncertain, dynamic environments, their performance in these environments
may be improved if provided with visualization tools that enable them to forecast and predict
future spatial-temporal states and their associated uncertainty.
5
Another rationale for placing the human in the role of estimating TOA uncertainty associated
with a TOA prediction is evident from Wickens! (2000) research on automation reliability. It
has been shown that, when the operator is unaware that an automated system is less-than-
perfectly reliable, performance decrements, such as slowed failure detection time and lower
situational awareness (SA), are observed. This occurs because human operators tend to
become complacent and overly trusting of the automated systems when they should be
monitoring operations, considering alternative hypotheses, or preparing for alternative
courses of action. When users are aware of automation unreliability, however, they tend to
calibrate their cognitive strategies with the actual level of imperfection (Wickens, 2000) and
exhibit improved allocation of visual attention to raw data, as manifested by their visual scan
patterns, ultimately yielding improved performance. Paradoxically, because an operator
may attend more to the raw data underlying an automated cue, or implement a wider scan
pattern when they are uncertain of the outcome, they may actually perform better than when
they are certain that the automation is perfectly reliable.
Endsley, Bolte & Jones (2003) proposed that asking operators to manage TOA uncertainty
explicitly may help operators maintain operator attention and situation awareness, and
actually alleviate some of the negative consequences frequently attributed to increasing
levels of automation. In essence, it may not be desirable to totally eliminate uncertainty from
the environment; rather, it is important for operators to maintain an accurate awareness of
the state of the world in which they are operating, and that includes an awareness of the
level of certainty of the world or system. Endsley et al. proposed that the relationship
between confidence and situation awareness (SA) plays a significant role in the quality of
decisions. If confidence is high, and SA is high, an operator is more likely to achieve a good
outcome. However, given the same level of SA, but lower levels of confidence, the operator
will be less likely to act, choosing to gather more information, and thus be ineffective. Lack
of confidence therefore can contribute to indecisiveness and delays and the operator in this
case may be less likely to revise a plan when appropriate. On the other hand, if an operator
with poor SA recognizes that it is poor (i.e., has low confidence) he will correctly choose not
to act or will continue to gather more information to improve SA, and avert a potentially bad
outcome. The worst situation is the operator who has poor SA, but high confidence in his
erroneous picture of the world. This operator will likely act boldly but inappropriately. A
critical issue, according to Endsley et al., is ensuring not only that operators have a good
6
picture of the situation, but also that they are able to maintain an appropriate amount of
confidence with respect to that picture. As such, it is very important that operators
understand the degree and nature of uncertainties in the system.
Endsley et al. discussed the relationship between situation awareness (SA) and decision
making (Endsley, Bolte & Jones, 2003), together with the task/system factors and individual
factors that influence both of them. They also showed that system capability, interface
design, operator stress, workload, task complexity, and degree of automation can influence
an operator!s SA, decisions, and actions. According to Endsley et al., SA comprises three
levels. Level 1 refers to perception of the elements of the current situation. In an aviation-
related TOA scenario, an operator would be considered as having good level 1 SA if he/she
knows the location of each aircraft, its speed, and distance to go. Level 2 refers to
comprehension of the current situation, or the ability of the operator to determine whether
the aircraft are ahead, behind, or on schedule. Level 3 refers to projection of future states,
or the ability for the operator to survey the airport surface and predict whether all of the
aircraft under his control will arrive at the final destination on time, early, or late, based on
potential traffic and/or route complexities.
1.1.4 Defining Uncertainty
Before embarking on research to determine how operators comprehend TOA uncertainty, it
is imperative to first define TOA uncertainty. There is no clear consensus in the literature as
to how to define or quantify uncertainty associated with a TOA prediction, or what factors
should be included in an uncertainty metric. As a starting point, Schunn, Kirschenbaum and
Trafton (2005) have produced a potentially useful taxonomy of informational uncertainty that
is arguably the most comprehensive to date. Informational uncertainty is defined as
objective uncertainty in the information a person uses to make decisions. Drawing on
expertise in three domains – weather forecasting, submarine / sonar operations, and
medical imaging – they defined four classes of informational uncertainty that are prevalent
and generalize across many complex domains and labelled them as Physics,
Computational, Visualization, and Cognitive uncertainty. The focus of this research is
cognitive uncertainty; however at the same time it is acknowledged that this interacts with
several other forms of uncertainty, such as future prediction uncertainty, and physical
7
uncertainty associated with aircraft dynamics and environmental factors.
Physics Uncertainty refers to uncertainty in the raw, measured data. This is further
subdivided into three sub-types: Absence of signal being measured, noise or bias in the
signal, and noise or bias introduced as the signal is transduced (or converted into data).
Computational Uncertainty refers to the computational procedures that are applied to data
before they are presented to the human operator. There are three ways that computational
procedures can add new sources of uncertainty. The first (and the focus of the present
research) is future prediction uncertainty. Data are collected and may be accurate at a
specific point in time; however, the world is dynamic and continues to change.
Computational procedures either make no corrections for these changes or make only
partially accurate corrections, and this introduces a potentially large source of uncertainty. A
second form of computational uncertainty is statistical artefact uncertainty, in which
statistical algorithms such as data smoothing introduce artefacts in the displayed data and
distort reality. A third type of computational uncertainty is that related to the use of fast and
cheap algorithms that present the current best guess so that the problem solver can make
quick (though approximate) judgements if necessary. The third category of uncertainty
information within the taxonomy is Visualization Uncertainty. Here, the presentation of data
in the form of a map, table, or graph creates uncertainty, either by failing to represent
relevant information or by presenting it in a misleading way. The final category is Cognitive
Uncertainty. Inclusion of this category in the taxonomy acknowledges that humans, in acting
as an encoding device, information storage/retrieval device, and procedure enactors, are a
key, but error-prone, part of the information system. As possible errors are introduced into
these tasks, sources of information uncertainty are introduced. Cognitive uncertainty can be
created because of perceptual errors, memory encoding errors, information overload,
retrieval errors, background knowledge errors, and skill errors1.
1.2 RESEARCH OBJECTIVE
Given that uncertainty is inherent in systems, it is important to consider how operators in
general comprehend such uncertainty, and to identify conditions under which an operator
1 Because skilled behaviour is conventionally modelled as being automatic, "skill errors" should arguably not be
included as a type of cognitive error.
8
believes there to be more or less uncertainty than actually exists in the world. With regards
to TOA uncertainty in particular, an improved understanding of how operators comprehend
this uncertainty and the biases inherent in the process is critical to the development of
operations and technology in complex, uncertain, dynamic environments that involve
estimating time of arrival. A model of TOA uncertainty estimation is a critical first step
toward the development of user-centred DSSs. It is expected that the findings of this
research will extend to operators who make decisions in a variety of complex, dynamic, and
uncertain environments – i.e., not only aviation, but also process control, surface
transportation, and military command and control domains.
9
CHAPTER 2: EXPERIMENT 1 – AN INVESTIGATION AND PRELIMINARY MODEL OF
TOA UNCERTAINTY IN AVIATION SURFACE (TAXI) OPERATIONS
2.1 AVIATION SURFACE (TAXI) OPERATIONS
The concept of TOA uncertainty is applicable to many domains; however, it is helpful to work
in a single problem domain in order to ground the problem in reality. The chosen domain is
that of airport surface (taxi) operations. In current day airport surface operations, a
controller is responsible for both issuing taxi clearances and monitoring aircraft for
conformance to the issued taxi route. Controllers monitor for potential conflicts and act in a
tactical / reactive manner by issuing a hold command to aircraft when needed to prevent
conflicts. The ability of a controller to predict the time of arrival of an aircraft is limited, mostly
because the controller lacks the tools and information needed to make such predictions.
This leads to ultra-conservative, risk averse behaviour, as exhibited by the tendency for
controllers to require larger than necessary separations between aircraft. For example, when
clearing taxiing aircraft to cross an active runway, the controllers tend to build a queue of
several aircraft to cross the runway all at once during a single large break in arriving traffic,
rather than attempting to sequence the taxiing aircraft in a coordinated fashion between
arriving aircraft. This ultimately reduces airport capacity and increases the delay
experienced by passengers on the airport surface, with the first aircraft in the queue often
waiting 20 minutes or more. Furthermore, an aircraft that is stopped requires twice as long to
!spool up" and cross a taxiway/runway than an aircraft that arrives !just in time" without
stopping. This effectively doubles runway occupancy time and reduces arrival and
departure throughput. Upon arrival at its destination (departure runway or gate), an aircraft
will often have to stop and wait (either for take-off or for the gate to become available). This
creates inefficiencies, as a stopped aircraft may block a gateway, a taxiway, or runway exit
and restrict movements.
The future aviation system is moving to 4-D operations2, which will involve the issuance of a
time- or speed-based clearance by the controller, with the additional requirement to monitor
2 The term "4-D operations" is currently used in the aviation industry in reference to the time/speed
element; however, in airport surface (taxi) operations, the third dimension, altitude, is of course constant.
10
for TOA conformance. It is expected that pilots will be required to arrive at the runway (for
crossing or take-off) within a specified TOA window. The acceptable duration of this
window, and therefore the precision requirements, have not yet been determined.
Currently, automated systems are being developed to assist the controller in issuing
commanded TOA, and also to predict aircraft TOA, but these predictions will be inherently
uncertain, given the dynamic nature of the environment. Pilots! taxiing ability, weather,
traffic, and route complexity are among the factors that may cause an aircraft to alter its
speed and thus may influence its probability of arriving on time. The degree to which
controllers can effectively estimate TOA uncertainty will impact their situational assessment
and decision-making abilities, yielding safer, more efficient operations.
As a first step in creating a model of how ground controllers actually deal with TOA
uncertainty in their environment, experiment 1 was carried out with expert ground
controllers. The goal of this experiment was to understand factors that are relevant to
estimating TOA uncertainty in real-world settings and which should be considered for further
evaluation in future studies. A second goal was to gain an understanding of the task of TOA
uncertainty estimation, as is characterized in a preliminary process model of TOA
uncertainty estimation later in this chapter, and evaluated in more rigorous experimental
studies presented in Chapters 4 and 5.
2.2 EXPERIMENT WITH AIR TRAFFIC CONTROL (ATC) SUBJECT MATTER EXPERTS
This experiment explored factors that contribute to air traffic controllers! certainty that a time-
based taxi plan issued to pilots would be carried out as scheduled. The experiment was
conducted in two parts: First a desktop simulation experiment was conducted to collect
subjective ratings, given a number of factors believed to influence TOA uncertainty. Second,
the same controllers participated in a debrief interview to further understand the factors that
contribute to TOA uncertainty in actual current-day and potential future time-based taxi
operations.
11
2.2.1 Method
2.2.1.1 Participants
One current and three recently retired air traffic controllers with extensive major airport tower
experience were recruited. Their mean age was 56.75 years and their mean years of
experience at major international and domestic airports was 23.5. All participants had
worked in local controller and ground controller positions, while two of the participants had
additional experience in TRACON (tower control). They were all familiar with tower
operations and procedures and common tower radar and computer systems.
2.2.1.2 Apparatus
Static pictures of typical taxi routes (as shown in Figure 2-1) at a complex airport, Dallas
Forth Worth (DFW), were presented using SuperLab software on a desktop Macintosh G3
computer with a 23” Apple cinema display. Each picture depicted a taxi route (highlighted in
magenta) and an aircraft icon (yellow triangle) that showed the aircraft!s position along the
route. The time that had elapsed since the aircraft started the route was shown in a white
box beside the aircraft icon (shown as 2 minutes in Figure 2-1). Each taxi route ended at a
runway that was highlighted in yellow. An assigned runway crossing time window (a window
of time within which the aircraft had to reach the runway threshold) was presented in a
yellow box beside the runway crossing point (shown as 5:41 to 6:11 in Figure 2-1).
2.2.2 Procedure
2.2.2.1 Introduction
The instructional protocol for this experiment is presented in Appendix A1. The concepts of
coordinated runway crossings and time-based taxi clearances were explained to each
participant, as well as the rationale in terms of both system-wide safety and efficiency
benefits. Participants were told that, if an aircraft arrives early at the runway-to-be-crossed,
the aircraft will have to stop and wait for the assigned crossing time, which doubles the time
required for an aircraft to cross a runway due to engine spool-up times. Furthermore,
arriving late will mean that the aircraft has missed the crossing window and this will
inevitably cause a delay.
12
Figure 2-1. Experiment 1 stimulus: Dallas Fort Worth (DFW) Airport layout, shown with an aircraft (yellow triangle) taxiing along a taxi route (magenta path) en route to a to-be-crossed runway (yellow highlight). Elapsed time is shown in the white box and assigned runway-crossing window is presented in the yellow box.
2.2.2.2 Distance/Speed Calibration
In order to calibrate the air traffic controllers to the approximate times required to taxi along
different routes at DFW, participants were shown four different taxi routes overlaid on the
DFW airport (as in Figure 2-1 above). Two routes (6000! and 12,000!) running east/west and
two running north/south across the airport were displayed. The time to complete each route
at three different speeds (16 kts, 20 kts, and 24 kts) were also provided, assuming constant
speed, as shown in Table 2-1, along with the graphical route depictions. The controllers
were instructed that typical taxi speeds ranged from 16 to 24 kts, with an average speed of
20 kts.
13
Table 2-1. Distance and TOA for each route distance and speed combination.
Time to complete route per route distance (in Minutes:Seconds)
Taxi Speed
6,000! 12,000!
16 kts 3:42 7:24
20 kts 2:58 5:56
24 kts 2:28 4:56
2.2.2.3 Experimental Trials
Subjects were shown 162 static pictures of taxi plans in progress as in Figure 2-1. Each
picture showed the taxi plan and an aircraft!s progress in carrying out the plan – both its
current location along the route and the elapsed time since the aircraft started taxiing. Each
picture was a combination of the five independent variables presented in Table 2-2: route
distance, route complexity, runway crossing window size, position along the route, and
speed. Route Distance was expected to influence TOA uncertainty estimations, with longer
routes producing lower TOA certainty than shorter routes. Route complexity was
represented as the number of turns required to complete the route. Turns add variability to
the taxi speed, as pilots must reduce speed to safely manoeuvre the turn and accelerate
after the turn to resume taxi speed. It was expected that the increased speed variability
would increase TOA uncertainty. The duration of the runway-crossing window was a
manipulation of the degree of precision required by the taxiing pilot. It was expected that
controllers! TOA uncertainty would be higher when the aircraft was required to make a very
small crossing window than for a larger crossing window. The aircraft!s position along the
route was also manipulated. It was expected that TOA uncertainty would be larger for
aircraft at the start of their route than for those that had progressed appropriately along their
route towards their destination. The final independent variable was the nominal taxi speed
of the aircraft.
The subjects! task was to assess the information available in the depicted taxi plan and
provide a rating of how certain he/she was that the aircraft would arrive at its destination
within the allotted runway crossing window – neither too early, nor too late. Following the
sizeable literature (e.g., see Soll & Klayman, 2004; Block & Harper, 1991) that use subjects!
14
ratings of certainty as a proxy for estimating probability, subjects in this experiment provided
ratings on a seven-point scale from 1 (Very Low Certainty) to 7 (Very High Certainty) by
entering a number on a standard keyboard, and were encouraged to use the entire range of
the seven-point scale. Subjects were told to assume that it was a clear day at DFW airport
and that no traffic conflicts were anticipated. Also, they were told that the aircraft represent
a Boeing 757 that taxis at 20 kts on average but that any number of factors could influence
their ability to maintain a constant 20 kts.
Table 2-2. Independent Variables
Independent Variable Levels
Route Distance 6000!, 12000!
Route Complexity 1, 2, 3 turns per 6000!
Runway Crossing Window Size 15, 30, 45 seconds
Position along route Start, 1/3rd, 2/3rd
Taxi Speed 16, 20, 24 kts
Subjects first completed 18 practice trials that exposed them to each level of the five factors.
Then, after a short break, subjects completed 162 experimental trials comprising each
combination of the five factors. The order of the 162 trials was randomized, and they were
presented in 4 blocks, each separated by a short break. The task was self-paced.
2.2.2.4 Post-study Debrief
Upon completion of the experimental trials, a semi-structured interview was conducted with
each of the ground controllers to better understand their current tasks, how these might
change with the introduction of 4-D clearances, and their perceptions of TOA uncertainty on
the airport surface. Specifically, the purpose of the interviews was to explore factors that
contributed to their perceptions of certainty that a time-based taxi plan once issued to a pilot
would be carried out as scheduled. The interview questions are presented in Appendix A2.
2.2.3 Results
Prior to presenting the results of this experiment, is it noteworthy to recall that the main goal
of this experiment was to determine which factors are relevant to the task of estimating TOA
15
uncertainty, for the purposes of developing a model of the TOA uncertainty estimation task
for future empirical evaluation.
2.2.3.1 TOA Certainty Ratings
Recall that the dependent measure was a subjective rating of TOA certainty, with lower
numbers representing low certainty, and higher numbers representing higher certainty. The
mean and standard error of the ratings for each level of the five independent variables are
presented in Table 2-3. A Distance (3) x Complexity (3) x Runway window crossing size (3)
x Position (3) x Speed (3) repeated measures analysis of variance (ANOVA) was conducted.
Given the exploratory nature of this experiment, an alpha level of 0.1 was used in
interpreting these results.
There was some evidence that, as route complexity (number of turns) increased, the
subjects! TOA certainty ratings decreased, F(2,4)=5.00, p=.08. In particular, the high
complexity condition (3 turns) yielded lower certainty scores than both low (1 turn) and
medium complexity (2 turns; p<.05). Recall that route complexity represents speed
variability, thus this suggests that factors that contribute to speed variability on the airport
surface may reduce TOA certainty. (These factors were explored further in the debrief
interviews, as discussed in section 2.2.2).
Table 2-3. Certainty Ratings Mean and Standard Error
Independent Variable Condition Mean Rating
Standard Error
Route Distance 6000!
12000! 4.9 5.2
.627
.415
Route Complexity 1 turn 2 turn 3 turn
5.3 5.2 4.7
.367
.602
.566
Runway Crossing Window 15 sec 30 sec 45 sec
4.9 5.1 5.1
.496
.534
.502
Aircraft Position Start 1/3rd 2/3rd
5.3 4.6 5.3
.617
.406
.555
Speed 16 20 24
5.5 5.1 4.5
.388
.424
.730
16
There was a small, but significant, increase in subjectively rated TOA certainty as a function
of runway crossing window size, F(2,4) = 15.45, p=.013, with lower certainty ratings for the
15 sec window than both the 30 and 45 second window. This finding suggests a possible
threshold level associated with the required TOA precision at which the subjects! certainty
associated with an aircrafts! TOA may drop.
Speed yielded interesting results, with subjects! certainty scores decreasing as speed
increases, F(2,4)=5.829, p=.065. Comments made by the subjects during the debrief
interview suggested that certainty scores were lower under high-speed conditions because
they were unsure if the aircraft could maintain the high speed throughout the entire route,
especially given the required turns in the route.
Although there was no significant effect of route distance, there was a significant effect of
the aircraft position along the route on the subjects! certainty ratings, F (2,4)=4.84, p=.086.
As can be seen in Table 2-3, when the aircraft was 1/3rd of the way along the route, certainty
ratings were lower than when the aircraft was either at the start or 2/3rd along the route.
This suggests that the most uncertainty was perceived when the aircraft had not travelled
very far (thus the ability to assess mean speed was limited) and when there was a longer
distance remaining (thus the potential for increased variability in the speed on the route
remaining).
In summary, turn complexity, or factors expected to increase speed variability reduced the
subjects! certainty ratings, as did aircraft speed, short runway window crossing sizes (or high
TOA precision requirements), and unknowns regarding aircrafts! speed variability when only
a short distance has been travelled and larger distance remained. These results indicate
that future empirical studies should manipulate speed variability and distance remaining.
Related, feedback from the SMEs interviewed after this experiment revealed the need for
dynamic simulations so that subjects could better understand the amount of speed variability
for a given aircraft. Showing the aircraft position along the route with elapsed time (see
Figure 2-1) provided no indication as to whether the aircraft!s speed was constant or
variable.
17
Also, while the precision required for an aircraft to arrive at its destination (runway crossing
window) affected the subjects! ratings of certainty, results from the debrief interviews
suggested that this was a coarse level of granularity. Future studies should consider using
TOA estimation as a dependent variable rather than an independent variable – that is, rather
than asking subjects to rate their certainty associated with the aircraft arriving within a
certain time window, future studies should ask subjects to estimate the TOA window in
which they are certain the aircraft will arrive.
2.2.3.2 Interviews
The post-study interviews with the air traffic controllers were useful to provide insights into
the ground controller!s task. First, it is clear that, at any given time, a controller is
responsible for monitoring several aircraft simultaneously, often more than 20 aircraft, each
with its own starting point, destination, and route. Second, it was noted that, in time-metered
situations, it is generally beneficial for a controller to detect an aircraft that will arrive earlier
or later than the commanded time, so that the controller can issue a new speed command,
reroute the aircraft, or reroute another aircraft. Third, the uncertainty associated with on-
time arrival is due primarily to the variability of speed of the aircraft, which is determined by
the pilot in control of the aircraft. An important assumption for future operations is that each
pilot will be made aware of the optimal speed to travel and uncertainty with regard to the
TOA will be based on the pilot!s ability to maintain that speed.
The debrief interviews from the four air traffic controllers were synthesized into three
relevant topic areas, as discussed next.
1. How controllers determine TOA and TOA uncertainty
At the outset of this experiment, it was assumed that air traffic controllers calculate time of
arrival, albeit with rough approximations, by considering approximate route distances and
aircraft speeds. However, results of the debrief clearly suggest otherwise. Instead, it was
revealed that controllers develop heuristics to determine how long a taxiing aircraft should
take to complete a given route. These heuristics are built over years of on-the-job
experience. For example, the controllers stated:
18
• “I never thought about how long a 12000! route at SFO takes. I really don!t
know. I just know by experience how long it takes to get from one side of the
airport to the other”
• “Today, I found myself using math calculations [to do the experimental task]. I
would never do this in the real world.”
• “With experience at an airport, you just know how long it takes to travel a taxi
route”
These TOA heuristics developed by the controllers take a number of factors into
consideration, including the airport layout, aircraft type, airline culture, and traffic flow. For
example, controllers had developed expectations that pilots of a certain airline could
consistently be relied on to make an expedited runway crossing, whereas for others they
would take a different, more conservative approach, because the controller was uncertain
whether those pilots would comply. They cited issues such as corporate culture, experience
at the airport, and aircraft type as factors in their decision. The following are direct quotes
from the air traffic controllers selected to shed light on this process:
• “Pilots flying out of their own base [airport] are very familiar with airport
operations. They know the system and I can be more certain of their TOA.
They are easier than the "once a month! pilots, who tend to be more cautious
and apprehensive”.
• “We [ATC] have running jokes about airlines and which ones will comply to
expedited clearances. Some we know and others … maybe they will, maybe
they won!t”
• “I never told [Airline X] to do anything [with a time element to it] but [Airline Y]
would always be able to do it”
2. Sources of TOA uncertainty
The experiment reported above considered four factors that were expected to contribute to a
controller!s uncertainty of an aircraft!s TOA: route distance, route complexity, runway-
crossing window size, and position along the route. One of the findings of that experiment
was that speed variability, as manipulated by route complexity, was a large contributor to
TOA uncertainty. This finding was echoed by the controllers in the debrief interviews. One
19
pilot stated “It is the human element that is the big problem. The pilot has to be comfortable
with the visibility, the equipment, and the taxi route/ airport layout. The pilot has the final
say”. Other factors, in addition to those tested as independent variables in experiment 1,
that contribute to speed variability, and that controllers consider in estimating TOA
uncertainty include:
• Traffic patterns/ volume
• Aircraft type
• Weather
• Visibility
• Airline (familiarity with airport, past experience)
• Arrival traffic, separation on final
• Departure queue
• Flow control
• Aircraft equipage (with speed/time displays)
3. Effect of TOA uncertainty on ATC tasks
As aviation operations move from current-day operations to future time-based operations, it
is only reasonable to expect a change to ATC tasks and the way controllers operate. When
asked during the debrief interviews, each controller expressed concern about a need for
increased monitoring. For example, one controller stated that “[in cases of high TOA
uncertainty] controllers would have to monitor the aircraft more closely, and more frequently.
It may require re-routing to ensure conflict-free routing”. Similarly, another controller stated
that “it will change the way controllers monitor”, and also added that this change in
operations would mandate new displays and technologies to support the task. “We would
need more information, such as expected TOA and speed readouts, to know who!s not
going to make their commanded arrival time. A speed readout would be important, so that
we can see if the speed is reasonable and how much it fluctuates”
2.3 A MODEL OF TOA UNCERTAINTY ESTIMATION
Based on the results of this first experiment and its debrief interviews, a preliminary model of
the TOA uncertainty estimation task is proposed. This model will be used to guide
subsequent experiments. It is proposed that the process of estimating TOA uncertainty is a
20
two-step process, as depicted in Figure 2-2 and described next. Controllers, through years
of experience, develop a library of speed profiles that describe vehicles or situations that
they encounter, based on particular airlines, traffic patterns, or weather conditions. Each
speed profile consists of both an average speed and amount of speed variability (for
example, “this aircraft tends to be fast, but very variable – or consistently slow”). This speed
profile may be generated based on previous experience and expectations developed over
time and exposure, real-time observation, or training. At one extreme, if the variability is
zero, the aircraft will travel at a constant speed, there is no uncertainty in TOA, and the
operator should be able to predict TOA perfectly. At the other extreme, if the aircraft speed
is very variable, the uncertainty will be high, and the controller cannot easily predict TOA.
Figure 2-2. A model of the TOA uncertainty estimation process
During operations, controllers determine if each vehicle matches a known speed profile in
their personal library. For example, debrief comments suggested that controllers considered
all aircraft from a particular airline as sharing the same profile. Generally, airlines that flew
into an airport very infrequently were considered to have a speed profile that was slow and
highly variable, whereas those airlines that were based out of that airport were faster and
more consistent.
21
Operators then use knowledge of the speed profile (e.g., fast and variable) combined with
an estimate of the distance to travel, and an associated estimate of TOA and TOA
uncertainty to make decisions and act on those decisions, depending on factors such as
personal risk tolerances and consequences of error. These decision-making processes,
however, are beyond the scope of this research effort.
The model must also anticipate the situation in which the operator is faced with a new
vehicle profile with which he/she is not familiar. In this case, it is expected that the controller
would apply a heuristic to estimate TOA uncertainty for unknown profiles, based on their
similarity to known profiles. This was suggested by the controllers! comments, that
suggested a tendency to lump all aircraft from a single airline together – even though each is
piloted by a different pilot, whom they may or may not have encountered before.
2.4 SUMMARY
Experiment 1 resulted in a simple model of TOA uncertainty based on insights gathered
from skilled professionals who engage in TOA uncertainty tasks. The next step in the
process was to evaluate the model using empirical human-in-the-loop experiments. To
accomplish this, first an appropriate experimental method and platform was developed, as is
discussed next in Chapter 3. Two additional experiments, presented in Chapters 4 and 5,
were conducted to evaluate the above TOA uncertainty model, and a refined model is
presented and discussed in Chapter 6.
22
CHAPTER 3: METHODS AND METRICS TO EVALUATE
TOA UNCERTAINTY ESTIMATION
No literature was identified to date that has attempted to develop experimental methods and
metrics to assess an operator!s comprehension of TOA uncertainty. Indeed, there is a
small body of human factors literature that has investigated uncertainty displays. However,
much of that work was conducted to evaluate the effectiveness of a specific uncertainty
display for a domain-specific need (e.g. Andre & Cutler, 1998; Banbury et al., 1998) and
thus the generalizability of those findings, and methods, is limited. Other researchers, such
as Schaefer, Gizdavu & Nicholls (2004) have evaluated the usability of uncertainty displays
and assessed the impact of the uncertainty display characteristics on operator workload and
situation awareness. Just as with the subjective certainty ratings used in Experiment 1 of
the current research effort (reported in Chapter 2), those studies provided valuable insights,
but they are not sufficient to further our knowledge of the understanding and management of
uncertainty.
To objectively evaluate TOA uncertainty, methods and metrics that compare objective
uncertainty to subjective uncertainty are necessary. Objective uncertainty is a property of
the system under study, in this case derived from the dynamics of the aircraft and other
environmental factors. As such, objective uncertainty can be objectively computed, given
knowledge of the variability in the speeds of the aircraft. On the other hand, subjective
uncertainty refers to estimates by an observer of the system given his/her state of
information at the time. When perfectly calibrated, subjective uncertainty equals objective
uncertainty. On the other hand, if the observer is not well calibrated, he/she will either
overestimate or underestimate TOA uncertainty. Thus, the first challenge of the present
research is to develop an experimental task and metric that will enable evaluation of
uncertainty calibration.
3.1 ABSTRACTING DISTRIBUTIONAL PROPERTIES
The ability to learn the distributional properties of probabilistic information is critical to the
task of estimating TOA uncertainty. In daily experience with probabilistic processes, it is
very rare that we are told explicitly what an underlying distribution is – i.e., whether it is
normal or skewed, or what are the mean and standard deviation that characterize the
23
distribution (Pitz, 1980). In these cases, it is necessary for an observer to infer properties of
the population from information that is contained in the samples of objects that they observe.
Such tasks require a person to engage in !abstraction," or the process of incorporating into
the representational structure information that has not been presented directly (Pitz, 1980).
Pitz used the term !prototype" to refer to the structure that describes the representation of
information about a population that is inferred from sample information. A prototype is a
theoretical concept that has been employed to describe the end product of abstraction.
Although it is still largely an open question what properties of the population might be
abstracted from a sample, it is known that the process of inferring an average value for
uncertain quantities is a very general process, and one that humans tend to do quite well
(Pitz, 1980; Peterson and Beach, 1967). On the other hand, humans have more difficulty
estimating statistical variance, while skewness and bimodality of a distribution were
recognized and used by the subjects in making their predictions. Pitz concluded that
information about central tendency is abstracted as a prototype; however it appeared that
certain critical features of the sample information are stored directly – such as the extreme
values, the smallest and the largest that have been observed.
Peterson and Beach (1967) found that estimates of variance tend to be influenced by the
mean value of the stimuli. They provide the analogy of comparing the smoothness of the
surface of a forest to that of a desktop. The treetops seem to form a fairly smooth surface,
however, the items on the surface of a desktop would seem very bumpy and variable. This
forest top is far more variable than the surface of the desk, but not relative to the sizes of the
objects being considered. They concluded that, instead of estimating variance, subjects tend
to estimate the coefficient of the variation (the standard deviation divided by the mean).
The issue of how a person represents knowledge about variability and skewness is less
clear. Pitz (1980) proposed that it is possible that measures of variability are inferred and
stored as part of the prototype, along with information about the central tendency. Estimates
of variability can be generated by finding the (absolute) difference between each value and
the currently estimated average value, and taking the average of these differences.
However, such a procedure would place a fairly heavy load on short-term memory and is
unlikely to occur as a spontaneous strategy. Alternatively, it is possible that people use
24
information about the central tendency together with their knowledge of the extreme values.
The difference between the two extremes provides information about variability – while the
location of the central tendency relative to the extremes provides information about
skewness.
Related to this is the issue of how human operators factor in extreme values and outliers.
Alpert and Raiffa (1982) found that subjects were unwilling to weight large deviations
heavily. They gave five groups of subjects almanac questions. All subjects were asked to
report 25th, 50th, and 75th percentiles. In addition, Group 1 was asked to report 1st and 99th
percentiles; Group 2 was asked to report 0.1th and 99.9th percentiles; Group 3 was asked to
report “minimum" and “maximum" values; and Group 4 was asked to report “astonishingly
low" and “astonishingly high" values. In every case, the estimated spread of the tails of the
distributions was too small, regardless of the definition of the extremes and, although
feedback did improve the spread, it did not completely eliminate the overconfidence bias.3
O!Connor and Lawrence (1989) utilized a task that involved time series predictions and they
found that calibration of subjects! estimated confidence intervals was influenced by the
degree of forecasting difficulty. For simple series, subjects were underconfident (that is, the
subjects! estimated intervals were too large), but for medium to high difficulty series, the
subjects were overconfident (the estimated intervals were too small).
The research discussed above evaluated operators! ability to abstract mean and variance
information of a population from samples of numbers. However, the ability to abstract the
mean and variance of a series of times of arrival (TOAs) has evidently not been considered
in the literature, in spite of the fact that understanding human capability to abstract TOA
information from observing a sample of multiple objects with variable speed is particularly
important to understanding how human operators comprehend TOA uncertainty. This
becomes a particularly interesting problem when one considers that TOA is a non-intuitive
metric, due to the fact that, because TOA varies inversely with speed for a given distance
3 In statistics, a confidence interval is an interval estimate of a particular population parameter. For example, a
95% confidence interval means that it is estimated that 95% of samples from the population will lie within this
interval. If a person’s subjective 95% confidence interval were to be narrower than the actual interval, that is
they underestimated the confidence interval, then it can be said that the subject is overconfident with respect to
the actual corresponding interval.
25
(T=D/V), TOA, by definition, is not a linear function of speed. Rather it exhibits a function as
depicted in Figure 3-1, which plots TOA as a function of speed for variables that are along
the order of those encountered in the ground controller!s world.
As can be seen in Figure 3-1, because TOA is the reciprocal of speed, slower speeds have
a bigger effect on TOA. That is, a difference of 1 mph at slow speeds (for example, an
increase from 1 to 2 mph) has a large impact on TOA (in this example, assuming a distance
of 1 mile, TOA at 1 mph = 1 hour, while TOA at 2 mph = .5 hour.) However a difference of 1
mph at a higher speed (for example, from 60 to 61 mph) changes the TOA by only .003
hours (TOA at 60 mph = .0167, while TOA at 61 mph = .0164 for a distance of 1 mile).
Related to this, if one considers a range of speeds that spans 5 to 15 mph, one can see in
Figure 3-1 that the resultant range of TOAs is much larger than if the same 10 mph range
were centred around a speed of 85 mph. It follows then that speeds selected from a normal
distribution of slower speeds would yield a more positively skewed4 distribution when
converted to TOAs than would speeds selected from a normal distribution of faster speeds.
Figure 3-1. The relationship between speed (mph) and TOA (hours) assuming distance = 1 mile
4 A positively skewed distribution is one whose tail is elongated to the right; that is, more data are encountered in
the right tail than would be expected in a symmetrical distribution.
!"
!#$"
!#%"
!#&"
!#'"
!#("
!#)"
!#*"
!#+"
!#,"
$"
!" $!" %!" &!" '!" (!" )!" *!" +!" ,!" $!!"
Tim
e o
f A
rriv
al (h
ours
)"
!"#$%&'()*+,-.)Speed (mph)
26
Furthermore, as the standard deviation (SD) associated with the speed distribution
increases, the positive skew of the resulting TOA distribution would be exaggerated. This
becomes clear if one considers that numbers drawn randomly from a distribution with a
larger SD will consist of more extreme speeds than if drawn from a distribution with a
smaller SD. This is demonstrated in Figure 3-2 that shows in the left panel, histograms
derived from three normal distributions of speed with the same mean (20 mph) and different
standard deviations (2, 4, and 8). The right panel shows histograms of the corresponding
times of arrival (hours) assuming a distance of 1 mile. As can be seen, the positive skew of
the TOA distribution increases considerably as the standard deviation increases from 2 to 8.
Also, consider Table 3-1, which shows the Mean, SD and skewness for distributions of TOA
that are defined by various combinations of means and standard deviations of speed.
Clearly the distribution is much more skewed (skew = 20.1) at lower speeds (M = 20) with a
high standard deviation (SD = 8), than for higher speeds (M = 60) with the same standard
deviation, in which case the skew was 1.2.
Table 3-1. Effect of mean and standard deviation of speed on TOA
Speed Distributions
TOA M 20 SD 2
M 20 SD 4
M 20 SD 8
M 30 SD 2
M 30 SD 4
M 30 SD 8
M 60 SD 2
M 60 SD 4
M 60 SD 8
M .05 .05 .07 .03 .03 .04 .02 .02 .02
SD .005 .01 .23 .002 .005 .013 .0006 .0011 .0025
SKEW .9 2.5 20.1 .5 1.2 3.9 .4 .6 1.2
Notes: For each speed distribution (defined by a Mean and SD of Speed), 1000 samples were drawn using the Regress+ software package, which possesses a normal distribution random number generator. Each speed was converted to a TOA assuming a distance of 1 (i.e., TOA = 1 / speed) yielding 1000 TOAs for each combination of Mean and SD of speed. Descriptive statistics (mean, SD, and skew) were calculated for the 1000 TOAs of each speed distribution condition.
27
Figure 3-2. Left panel shows three histograms of speeds drawn from a normal distribution with mean speed 20 and (top to bottom) SD=2, 4, or 8. Right panel shows corresponding TOA distributions with increasingly positive skew as standard deviation increases.
!"#$%&%'(%
)*%&%'%
)+,-%&%(%
!"#$%&%.(/0%
)*%&%.((/%
)+,-%&%.1%
!"#$%&%'(%
)*%&%2%
)+,-%&%(%
!"#$%&%.(/'%
)*%&%.(0'%
)+,-%&%'./%
!"#$%&%'(%
)*%&%3%
)+,-%&%(%
!"#$%&%.(40%
)*%&%.(23%
)+,-%&%'(.0%
Speed Time of Arrival
%%%%%.('%%%%%%%%%%%%%%%%%%%.(2%%%%%%%%%%%%%%%%%%.(4%%%%%%%%%%%%%%%%%%.(3%%%%%%%%%%%%%%%%%%.0%%%%%%%%%%%%%%%%%%%.0'%
.(2%%%%%%%%%%%%%%%%%%%%%%%.(/%%%%%%%%%%%%%%%%%%%%%%%%.(4%%%%%%%%%%%%%%%%%%%%%%%%.(5%
%%%%%.((% .'%%%%%%%%%%%%%%%%%%%%%%%%%%%.2%%%%%%%%%%%%%%%%%%%%%%%%%%.4% %%%%%%%%%%%%%%%%%%%%%%%.3%%%%%%%%%%%%%%%
%%%%%%%%Velocity (mph) Time of Arrival (hours) %
Speed Time of Arrival
Speed (mph) Time of Arrival (hours)
Fre
qu
en
cy
Fre
qu
en
cy
Fre
qu
en
cy
Fre
qu
en
cy
Fre
qu
en
cy
Fre
qu
en
cy
28
3.2 EXPERIMENTAL METHODS AND METRICS
To evaluate the model of TOA uncertainty, experimental task and metrics are required that
capture the essence of the real-world task faced by air traffic controllers in time-based
environments. Based on the findings of the experiment reported in Chapter 2, the task
should possess the following characteristics.
1) Provide an opportunity for subjects to build speed profile models based on
experience. The air traffic controller interviews clearly revealed that, rather than
real-time calculations, controllers estimate TOA and TOA uncertainty based on
expectations, gained through on-the-job experience. To replicate this in a controlled
experiment requires dynamic simulations (as opposed to the static route plans used
in experiment 1 in Chapter 2) that allow subjects to build the experience needed to
generate estimation heuristics. Providing speed read-outs will also facilitate the
subjects! understanding of the vehicle!s speed profile and remove issues
associated with speed perception.5
2) Avoid confounds associated with subjects! previous experiences. Since each
controller has already built expectations based on their unique individual
experiences, using actual controllers and real-world ATC tasks will likely introduce
confounds with previous experience that cannot be experimentally controlled.
Thus, there is a need for experimental controls that dictate the subjects! experience,
rather than relying on the individual experiences that each subject may bring to the
experiment from their own real-world experience, as these are widely variable
across individuals. This will require that speed variability be modelled in a generic
way not tied to expectations of airport operations, airlines, traffic etc.
3) Consider the interaction among speed, distance, and time. To estimate TOA
and TOA uncertainty, air traffic controllers consider both speed and distance.
5 The implications of this premise are discussed in Chapter 7. While issues related to velocity perception are an
interesting aspect of the TOA estimation problem, relying on velocity perception rather than velocity read-outs
would have the potential of confounding the main experimental findings related to how subjects estimate TOA
uncertainty. Furthermore, it can be safely assumed that accurate ground velocity readouts will be available to
both pilots and ATC in future aviation operations.
29
Furthermore, there are two factors associated with vehicle speed that must be
considered: average speed and speed variability. Inappropriate estimates of TOA
or TOA uncertainty can result from misestimating either speed or distance. This
must be addressed, either by controlling (holding constant) one or more factors or
experimentally manipulating each factor.
4) Approximate ATC workload. While it is not believed to be necessary to replicate
the entire ATC experience by including all task elements in high fidelity, it was
determined that approximating the visual monitoring workload was important. The
interviews with the ATC subjects suggested that ground controllers at major airports
are regularly required to monitor approximately 20 aircraft at a time.
5) Include known and unknown profiles. Controllers must work with both pilots and
airlines that fly into their airports regularly as their base operations and those that
fly in very infrequently. This means that matching the observed speed profile to
those in their library of profiles will be more difficult for those vehicles with
infrequent exposure. The experimental method should address both cases.
3.2.1 Experimental Task
An experiment was designed to simulate the critical elements of the task of an air traffic
ground controller (or other operator), who is responsible for monitoring and predicting the
ability of an aircraft (or object) to arrive at a defined location at a specific time, or within an
allowable time window around a specific time. The task involved several objects, each
travelling along a separate path (with different lengths). All objects started at the same time
and shared a common commanded TOA. Each was assigned a mean speed (which was
computed based on the speed required for on-time arrival) and a standard deviation. Each
object changed speed at pre-determined intervals, with the new speed selected from a
normal distribution defined by the assigned mean and standard deviation. Based on this
distribution, the TOA uncertainty could be computed objectively. Given a symmetrical
distribution of speeds about the mean speed, the expected TOA of each aircraft will be
correct and will not change (that is, if each is given a mean speed at the outset, based on a
proper estimated TOA). However, the probability of arriving within the permitted time
window will change, depending on the variability of the speed.
30
3.2.2 Metrics to Evaluate Uncertainty Calibration
A metric is required that can be used to determine how well human operators can estimate
measures of central tendency and uncertainty for both speed and TOA of aircraft that move
with variable speed. The approach adopted within the current research was to use a
normative model, in order to identify factors relevant to the inference process for speed and
TOA. In particular, the approach was to examine the relationship between inferences made
by human observers and corresponding inferences that would be made by an optimal
observer who had knowledge of the underlying distribution. This approach is analogous to
the use of the ideal observer used in signal detection theory and !economic man" in
economics (Peterson and Beach, 1967). In this sense, Peterson and Beach argue,
probability theory and statistics fulfil a role similar to that of optics and acoustics in the study
of vision and hearing. Statistics can be used as a theory of the uncertain environment and
provide a basis for a descriptive theory of imperfect human inference (Peterson and Beach,
1967).
Peterson and Beach refer to two measures: accuracy and optimality. Accuracy is assessed
when subjects" estimates are compared to descriptive statistics of an observed sample. For
example, if subjects observe a sample of 20 aircraft, each with a speed drawn from a
population with a known distribution, accuracy would be assessed by comparing the
subjects" estimate of the mean speed of the 20 aircraft to the objective mean of the 20
aircraft speed samples. Optimality, on the other hand, is assessed when subjects"
estimates are compared to parameters associated with the underlying population from which
the sample is drawn (rather than with the computed sample estimates). That is, in
assessments of optimality, the subjects" estimates are compared to the actual mean of the
population distribution. The current experiments set out to assess the optimality of subjects"
estimates. Recall that the simple TOA model presented in chapter 2, based on input from
ATC subjects, posits that operators build experience over many exposures and develop a
library of TOA uncertainty expectations that they then apply, rather than assessing the
variability of each object individually. Thus, it is argued that assessing optimality is the more
appropriate variable to evaluate for the current research effort.
31
The optimality of subjects! estimates can be explored using any of a number of population
parameters, including measures of central tendency, variance, skewness, and confidence
intervals. Given the fairly consistent finding from existing literature that subjects perform
quite well at the task of inferring means, this was chosen as one measure of optimality for
this experiment, in order to determine if these past findings extend to dynamic stimuli and for
the variables of speed and TOA. Subjects! estimates of speed and expected TOA were
compared to the mean speed and TOA of the population distribution.
More relevant to understanding TOA uncertainty, however, is to assess the optimality of
subjects! estimates of TOA variability. No literature was identified that has attempted to
develop experimental methods or metrics to assess an operator!s understanding or
inference of TOA uncertainty. Nevertheless, we can draw upon the literature on interval
estimation for insights into this problem.
The fractile method (Block and Harper, 1991) is a widely used method (see for example:
Juslin, Wennerholm, Olsson, 1999; Soll & Klayman, 2004; McKenzie, Liersch, & Yaniv,
2008; Teigen, Halberg, & Fostervold, 2007) that requires participants to provide the smallest
intervals that include an existing true value with a pre-stated probability. The participant is
considered calibrated if the proportion of values that fall within the interval equals the given
probability. An example of a question that would be provided to a subject using the fractile
method is: “Give the boundaries of the interval for which you are 90% confident that it
encloses the number of chickens consumed by American households daily” (Juslin,
Winman, & Olsson, 2003). Although there are exceptions, the literature suggests that
subjects tend to be over-confident in their ratings, in that the proportion of true values
included by the subjects! estimated intervals is typically lower than the pre-stated probability
– that is, the intervals tend to be too small, such that less than the pre-stated probability of
the true values falls within the subjects! intervals.
However, subsequent studies have shown two factors that reduce this overconfidence: 1)
the format of the question and 2) the type of stimuli. Each is discussed next.
32
Format of Question.
Soll and Klayman (2004) conducted a series of three studies with the goal of determining
how to best elicit numerical estimates of intervals from subjects. They compared subjects!
estimates across three methods of interval elicitation:
1) Range interval; (e.g. “I am 80% sure Charles Dickens was born between 1750 and
1860”),
2) Two-point method, involving lower and upper boundary; (e.g. “I am 90% sure that
Dickens was born after 1750” and “I am 90% sure that Charles Dickens was born
before 1860”), and
3) Three-point method, involving a measure of central tendency (mean or median),
lower and upper boundary; (e.g. “I think that it is equally likely that Charles Dickens
was born before or after the year 1805” and “I am 90% sure that Dickens was born
after 1750” and “I am 90% sure that Charles Dickens was born before 1860”). 6
They found that the three-point method – asking for an estimate of central tendency in
addition to both a lower and upper boundary – yielded the lowest overconfidence and the
highest degree of calibration. Similarly, Block and Harper (1991) found that, when
participants were required to produce an explicit point-estimate first, before the lower and
upper boundary, overconfidence decreased. This was also the case when an external point-
estimate was provided by the experimenter. Paese and Sniezek (1991) concurred with this
finding by suggesting that, when the point-estimate is required first, the subject is required to
spend more time processing the task and the stimuli before assessing the probability
interval, and thus produces better-calibrated intervals.
Type of Stimuli.
The fractile method, when used with general knowledge content questions such as the
above example of estimating the interval that contains the daily consumption of chicken with
90% certainty, has been shown to generally produce over-confident ratings (Lichtenstein,
Fischhoff, & Phillips, 1982; Juslin, Wennerholm, & Olsson, 1999). Typically in those studies,
questions are asked of undergraduate students and are selected to be values that the
students are unlikely to know much about (McKenzie, Liersch, & Yaniv, 2008). Thus over-
6 Charles Dickens was born on Feb. 7, 1812.
33
confidence has been attributed to the use of such !general knowledge" questions for which
the subjects" judgment likely involves elaboration of general knowledge through inference
(Juslin, Winman, & Olsson, 2003). In contrast, tasks that require an assessment of sensory
input (e.g., which of two vertical lines is longer) do not show the same over-confidence bias
(Juslin, Winman, & Olsson, 2003). Erev, Wallsten, and Budescu (1994) also report that the
typical over-confidence bias is present when the uncertainty is internal to the subject (i.e.,
related to their knowledge), but not when the uncertainty is external (i.e., a stochastic
property of the environment). Furthermore, the overconfidence bias has been shown to
decrease or disappear when the objects of judgement are randomly sampled from a natural
environment (Gigerenzer, Hoffrage, & Kleinbolting, 1991; Juslin, 1994). Thus, it is believed
that this method is an appropriate method for use when subjects are asked to create
estimates based on an assessment of sensory input.
In the studies that follow, the three-point fractile method was used to elicit estimates of the
TOA interval. Subjects were asked to first provide an estimate of the TOA, and then provide
both a lower boundary and an upper boundary that would produce the smallest window that
they were 95% sure would contain the actual TOA. This measure was chosen because it
reflects both the standard deviation of the distribution and the skewness of the distribution.
The size of the subjects" TOA window was then compared to the objectively computed
intervals to assess the degree to which the human operators were calibrated to the objective
level of TOA uncertainty in the environment. Objective uncertainty was operationalized in the
studies that follow as the 95% confidence intervals around the mean speed or TOA. These
were compared to the subjective TOA window estimates to determine the degree of
calibration. As the variability (standard deviation) of the stimuli to be observed was
increased, so too should the uncertainty associated with it, as reflected by an expected
increase in the size of the estimated window.
The metric chosen to assess TOA uncertainty required that the subject estimate the lower
bound and the upper bound of the smallest time window within which they are 95%
confident the aircraft will arrive at its destination with a specified probability – for example, “I
am 95% confident that the aircraft will arrive between 1:55 and 2:05”. The goal was to
assign the smallest time window possible, while ensuring that the aircraft does not arrive
34
outside the window. An operator who is “optimally calibrated” will estimate time windows of
size equal to the objectively computed TOA windows, computed as the 95% CI of the normal
distribution. If the subject is not perfectly calibrated, he/she will either overestimate or
underestimate the duration of the subject!s TOA window. If the time window is overestimated
(that is, larger than would ordinarily be needed to be 95% certain), that suggests that the
operator is assuming a variance in TOA arrival times that is larger than the actual objective
variance, which therefore suggests that he/she is in fact underestimating the probability that
the aircraft will arrive within the objectively correct (95%) time window.
35
CHAPTER 4: EXPERIMENT 2 – THE EFFECT OF SPEED AND VARIABILITY ON
TOA UNCERTAINTY ESTIMATION
The goal of the present experiment was to determine how well human operators estimate
means and uncertainty for both speed and TOA of vehicles that move with variable speed.
Specifically, the goal was to determine how well subjects estimate the mean and 95%
Confidence Intervals of a population distribution based on observing a sample of 20 vehicles
drawn randomly from the population distribution. This research makes the assumption that
speeds are normally distributed.
4.1 SPECIFIC RESEARCH QUESTIONS AND HYPOTHESES
The specific research questions of interest, and their associated hypotheses, are:
1) How well do subjects estimate means of both speed and TOA (computed as the
difference between the objectively computed mean of the population distributions
and the subjects! estimate of the mean speed and TOA)?
Hypothesis 1: It was expected, based on the findings of Pitz (1980), that
subjects will estimate means with little estimation error; that is, no systematic
biases are expected.
2) How well do subjects estimate uncertainty associated with speed and TOA –
computed as the difference between the objectively computed 95% Confidence
Interval (CI) of the population mean and the subjects! estimate of the smallest
window that they are 95% sure will contain the average speed or actual TOA?
Hypothesis 2: Across all conditions, subjects! estimates of both speed and
TOA intervals will be smaller than objective 95% CIs. This is based on the
previously discussed research (e.g., Alpert & Raiffa, 1982; Juslin,
Wennerholm, & Olsson, 1999; Soll & Klayman, 2004; McKenzie, Liersch, &
Yaniv, 2008; Teigen, Halberg, & Fostervold, 2007) that suggests that human
decision-makers tend to be over-confident, and thus estimate less uncertainty
than actually exists.
3) Do the subjects! estimates differ as a function of the mean speed of the observed
stimuli?
36
Hypothesis 3: It is expected (based on Peterson and Beach, 1967) that the
difference between actual 95% CIs and subjects! estimated speed and TOA
windows will be larger for objects with slower speeds than for those with
faster speeds, due to the non-linear relationship between speed and time-of-
arrival (as shown in figure 3-1).
4) Do the subjects! estimates differ as a function of the standard deviation of the speed
of the observed stimuli?
Hypothesis 4: The difference between actual 95% CIs and subjects!
estimated speed and TOA windows will be larger for speeds with high
variability than for those with low variability. This is based on Alpert and
Raiffa!s (1982) findings that subjects inadequately account for "extreme!
values, of which there will be more in higher variability conditions.
5) Are subjects able to infer skewness in the TOA distribution and apply this
appropriately in their estimation of the TOA intervals?
Hypothesis 5: It was expected, based on Pitz (1980), that subjects will
assume a linear transformation from speed to TOA and therefore will
estimate TOA intervals that are symmetrical as opposed to the objectively
generated positively skewed CIs.
4.2 EXPERIMENTAL DESIGN
This experiment employed a fully-within-subjects experimental design with three
independent variables, summarised in Table 4-1: speed, variability, and repetition. The three
levels of speed7 (slow, moderate, and fast) map to 35 units/time, 45 units/time, and 75
units/time respectively. These speed levels were chosen based on three constraints:
1) Given the relationship between speed and TOA shown in Figure 3-1, an attempt was
made to span a wide range of speeds, in order to capture subjects! ability to estimate
TOA and TOA uncertainty at different points along the speed/TOA curve;
2) The speeds were chosen to yield three expected TOAs that were approximately
equally spaced;
7 While it may be more correct to refer to low and high speeds rather than slow and fast speeds, the latter term
were nevertheless used in the remainder of the thesis to differentiate the levels of speed (slow, moderate, and
fast) from the levels of variability (low, medium, and high).
37
3) Given concerns over the salience of extreme numbers, an attempt was made to
choose !like numbers"; specifically a decision was made to choose three speeds that
ended in !5". It was believed that the psychological difference between (for example) 35
and 37 is less than between 39 and 41.
The three levels of variability (low, medium, and high) map onto standard deviations of the
speed distributions. Specifically, the standard deviations chosen were 3, 6, and 9
respectively. These were chosen to ensure a wide range of speeds without producing
negative values, given the mean speed values chosen.
Each subject completed six repetitions of each speed x variability combination. This was
deemed a sufficient amount based on pilot testing, which revealed very little difference
among estimations across repetitions. Subjects completed six blocks of trials. Each block
consisted of one trial, representing each of the nine unique experimental conditions –
comprising each factorial combination of three levels of speed and three levels of variability.
The presentation order of the nine conditions within each block was randomized.
Table 4-1. Independent variables
Independent Variables Levels
Speed Slow, Moderate, Fast
Variability Low, Medium, High
Repetition 1, 2, 3, 4, 5, 6
4.3 METHOD
4.3.1 Subjects
Fifteen subjects were recruited from colleges and the community local to NASA Ames
Research Center, where the experiment was conducted. Subjects" ages ranged from 18 to
53, with the mean age being 35. The range of education levels possessed by the
participants included high school (3 participants), two-year college diploma (3 participants),
four-year university (7 participants), and graduate school (2 participants). Eight of the
subjects had taken at least one course in statistics. Prior to participation, all subjects
successfully completed a vision test to ensure that they could read text in the same font
style, size, and colour used for the experimental stimuli under lighting conditions that were
38
identical to those of the actual experimental trials. The experiment lasted approximately 2.5
hours and subjects were paid $10 an hour for their participation.
4.3.2 Materials
4.3.2.1 Apparatus
The experiment was conducted in a noise-proof, human-subject testing booth. Subjects
viewed a large screen monitor (32 in. x 18 in.) that subtended a visual angle of 60º from the
average viewing distance of 32 in. Subjects input their responses using a standard
keyboard and mouse. They were also provided a pen and paper and were told they could
use them to take notes during the experiment, if they wished. The monitor brightness and
resolution settings were held constant across all subjects.
4.3.2.2 Stimuli
Experimental stimuli, shown in Figure 4-1, consisted of 20 vehicles (shown as round circles
with a diameter of .5 in) travelling simultaneously from left to right along identical routes, with
one vehicle on each route. All vehicles started at the same time, at trial start. Each route
was 18 in. long and was divided into 10 equal length segments (but the segments were not
marked or visible to the participant). Each vehicle!s speed was constant during each
segment but changed from segment to segment according to pre-determined numbers that
were drawn from a normal distribution with a designated mean and standard deviation
according to the experimental condition. Data tags (1.5 in. x .25 in. in size) displayed the
current ground speed (GS) on a digital readout that updated at the beginning of every
segment. A digital clock, centred vertically, immediately to the right of the 10th vehicle,
displayed elapsed time, with a precision of one decimal point.
A numerical simulation was first conducted to create a population or parent distribution,
which was randomly sampled to select the speeds for any given vehicle. This allowed for
later determination of the 95% intervals, by selecting the 2.5 percentile and 97.5 percentile
from the parent distribution. The stimuli were created following the steps outlined below:
1. Using a random number generator (Regress+ software package) 250,000 speeds
were drawn randomly (with replacement) from a normal distribution with a given
mean and standard deviation.
39
2. The 250,000 speeds were then randomly assigned to 25,000 vehicles providing 10
speeds - one for each segment the vehicle would travel.
3. Assuming a segment distance of .05 miles, the elapsed time for each segment was
computed.
4. The 10 segment elapsed times were summed to provide a total route TOA for each
of the 25,000 vehicles. The 25,000 TOAs were sorted in ascending order, and then
binned into 20 equal groups each representing 5% of the TOAs.
5. For each sample of 20 vehicles, one TOA was randomly selected from each bin.
This ensured that the resulting sample of 20 vehicles was representative of the
distribution. This was necessary because of the relatively small sample size of 20
vehicles.
4.3.3 Procedure
4.3.3.1 Introduction and Training
Participants were welcomed to the test facility at NASA Ames Research Center. First, they
were asked to read the Instruction Protocol (Appendix B1) and then read and sign the
informed consent form. After completing a short demographic questionnaire (Appendix B2),
each participant completed a brief vision test using a standard Snellen eye chart (read from
a distance of 10 feet), and a computer-based vision test which presented ten three-digit
numbers on the experimental apparatus with the same font style, size, and colour used in
the experiment. Participants were asked to read the ten numbers out loud to verify that they
could read them without difficulty.
Next participants were shown the experimental apparatus, as described above, and the
experimenter verbally explained the purpose of the experiment, the experimental stimuli, and
specific details of the task to be performed, including how to input responses using the
keyboard and mouse, as described in the written instructional protocol. The participants
completed three practice trials while the experimenter watched. The three training trials
comprised the following Speed/Variability combinations:
Practice Trial 1: Slow/Low
Practice Trial 2: Moderate/Medium
Practice Trial 3: Fast/High
Participants were permitted to ask questions throughout the three practice trials.
40
Figure 4-1. Experimental stimuli shows 20 vehicle icons with current ground speed data tags. Actual route length was 18 in. Data tags showing ground speed (GS) were 1.5 in. X .25 in. Elapsed time was shown in a box on the right (shown as 21.1 above).
41
4.3.3.2 Experimental Trials Each trial consisted of twenty vehicles traversing routes of the same length. All twenty
vehicles shared the same speed profile – that is, their speeds were drawn from the same
population distribution. All vehicles started at the same time at trial start. After the last of the
20 vehicles completed the route, a question screen appeared, with the following question:
A vehicle!s speed profile is similar in nature to the vehicle you just observed and it is
about to traverse the same length route.
Please estimate this vehicle!s average (mean) speed.
Please provide the smallest window that you are 95% sure will contain the
vehicle!s actual average speed
Participants used a standard keypad to enter the average speed and then used the mouse
to manipulate window sliders to set the size of the window by setting the lower and upper
bounds. Once satisfied with these answers, the subject pressed !continue" and the second
query screen appeared with the following question:
A vehicle!s speed profile is similar in nature to the vehicle you just observed and it is
about to traverse the same length route.
Please estimate this vehicle!s time of arrival (TOA).
Please provide the smallest TOA window that you are 95% sure will contain
the vehicle!s actual TOA
Each participant responded in the same manner as above, and once completed, the
participant pressed !continue" to begin the next trial.
Participants completed six blocks of experimental trials, each consisting of 9 trials – one for
each combination of the three mean speeds and three speed standard deviations.
Participants took a short break after the second and the fourth block. The time to complete
each trial varied from 1 minute (in the shortest condition) to 5 minutes (in the longest
condition). The total duration of the experiment, including training, ranged from 2.5 hours to
3 hours.
4.3.3.3 Debrief and Follow-Up
Upon completion of the last trial, a verbal debrief session was conducted (see Appendix
B3). Participants were asked questions to ascertain the strategies that they developed,
42
whether they used memory aids, and whether they found one task (i.e. estimating speed or
time) easier than the other. The general purpose and objective of the experiment was
explained, the experimenter answered any questions, and the participant was thanked for
his/her time.
4.4 RESULTS
For each trial, subjects estimated the average speed, speed windows, TOA, and TOA
windows of a new !target" vehicle that shared the speed profile of the 20 vehicles that they
had just viewed. The subjects" estimate of the target vehicle"s speed and TOA was
compared to the objectively computed mean speed and TOA of the population distribution
from which the samples of 20 vehicles were drawn. The subjects" estimates of the smallest
window that they were 95% sure contained the vehicles" average speed (estimated speed
window) and TOA (estimated TOA window) were compared to the objectively computed
95% CIs (referred to as speed intervals and TOA intervals) of the population distribution
from which the samples of the 20 vehicle were drawn. The !actual" mean, intervals, and
skew for both speed and TOA are presented below in Table 4-2, and the method by which
the actual scores were calculated is presented in Appendix B4.
The following analyses were conducted:
1) Four separate analyses (3 x 3 x 6 Repeated Measures Analysis of Variance, ANOVA)
were conducted to determine the degree to which the subjects" estimates of speed,
speed windows, TOA, and TOA windows, differed as a function of the independent
variables: speed (3), variability (3), and repetition (6). These analyses were conducted
to assess whether subjects could differentiate among the chosen levels of speed and
variability. The inclusion of repetition in these analyses allowed for the determination of
the effect of learning, fatigue, or scenario differences across trials. Repetition was not
statistically significant in any analysis, suggesting that these effects had minimal
impact, if any, on the data and therefore will not be discussed further.
2) To determine if the subjects" estimates differed from the actual values, the 95% CI
around the mean of the subjects" estimates – of speed, speed windows, TOA, and TOA
windows – were analyzed to determine if the actual score was contained within the CI.
43
If the actual score was not contained within the CI, the subjects! estimates were
deemed to be significantly different than the actual score.
3) To determine if subjects were more likely to produce sub-optimal estimates as a
function of the independent variables, Estimation Error scores were calculated, by
subtracting the subjects! estimated value from the objectively computed, or actual,
value. A separate 3 x 3 x 6 repeated measures ANOVA was conducted on the absolute
values of the Estimation Error scores, to determine if subjects tend to make larger
estimation errors as a function of the independent variables speed, variability, and
repetition. Absolute values were used as these allow an assessment of estimation
errors regardless of direction – i.e. too fast and too slow, or too early and too late – in
light of the fact that, in the ATC operational setting under consideration, errors of both
kinds are detrimental to system performance. Again, there were no significant effects of
repetition and these results will not be discussed further.
4) To determine the degree to which subjects produce symmetrical or skewed estimated
TOA windows, a symmetry score was developed (defined below) and applied to both
the actual and estimated TOA intervals. The 95% CIs around the subjects! TOA
symmetry score was analyzed to determine if the actual symmetry score was contained
within the CI. Instances in which the actual symmetry score was not contained within
the 95% CI suggested that the subjects! estimated intervals were different in symmetry
than the actual intervals.
4.4.1 Estimated Speed
Table 4-3 presents the objectively computed (actual) mean speed and the size of the 95%
confidence intervals for each combination of speed and variability, the subjects! mean
estimated values for speed, and the mean size of their estimated speed windows. It also
includes the mean estimation error scores, calculated as the mean of the differences
between the actual and estimated values, for both mean and window size.
44
Table 4-2 Actual mean values, interval sizes, and skew for speed and TOA
Speed TOA Independent
Variables
Speed/Variability Mean
Lower Bound
Upper Bound
Interval Size
Skew Mean Lower Bound
Upper Bound
Interval Size
Skew
Slow/Low 35 33.2 36.9 3.7 .01 51.8 49.1 54.8 5.7 .14
Slow/Medium 35 31.3 38.7 7.4 .00 53.1 47.4 60.1 12.7 .42
Slow/High 35 29.4 40.6 11.2 .00 56.2 46.3 71.5 25.2 34.32
Moderate/Low 45 43.1 46.9 3.7 .00 40.2 38.6 41.9 3.4 .12
Moderate/Medium 45 41.3 48.7 7.4 .01 40.8 37.5 44.6 7.1 .28
Moderate/High 45 39.4 50.5 11.1 .00 41.9 36.6 48.8 12.3 .73
Fast /Low 75 73.1 76.9 3.8 .01 24.1 23.5 24.7 1.2 .08
Fast /Medium 75 71.3 78.7 7.4 .00 24.2 23 25.4 2.4 .17
Fast /High 75 69.4 80.5 11.1 .00 24.4 22.6 26.4 3.8 .47
First, an analysis was conducted to determine if the subjects! estimated mean speed differed
as a function of the experimental conditions. This was to verify that the task conditions
(speed and variability) had been perceived as being significantly different. Subjects!
estimated mean speed, shown as a dashed line in Figure 4-2, did indeed differ as a function
of speed (F (2,28) = 7306.75, p < .001). Post-hoc paired t-tests revealed significant
differences among all three levels of Speed (p < .001 for all pairwise tests), providing
evidence that subjects were able to differentiate among the three levels of speed. There
were no differences in the subjects! estimates of mean speed as a function of variability or
repetition.
Next, the subjects! estimated mean speed was compared to the actual mean speed by
examining the 95% CI around subjects! estimated mean speed to determine if the 95% CI
contained the actual mean speed. Figure 4-2 presents both the actual mean speed and the
45
subjects! estimated mean speed, the latter plotted with error bars which reflect the 95% CI
around the mean. In conditions in which the 95% CI did not contain the actual speed, it was
concluded that the subjects! estimates were significantly different than the actual speed.
This was the case for three conditions: moderate speed/medium variability, high
speed/medium variability, and high speed/high variability. In each of these three cases, the
actual speed was greater than the estimated values, revealing that subjects underestimated
speed in these three conditions.
Table 4-3. Actual and estimated speed means, intervals, and estimation errors
Actual Speed Subjects! Estimated
Speed
Estimation Error (Actual –
Estimated) Independent Variables (Speed/Variability)
Mean 95% CI Mean Window
Size Mean
Window Size
Slow/Low 35 3.7 34.7 6.0 .3 -2.3
Slow/Medium 35 7.4 35.1 9.0 -.1 -1.6
Slow/High 35 11.2 34.2 12.6 .8 -1.4
Moderate/Low 45 3.7 44.4 6.0 .6 -2.3
Moderate/Medium 45 7.4 43.4 8.8 1.6 -1.4
Moderate/High 45 11.1 44.4 11.9 .6 -.8
Fast /Low 75 3.8 74.7 5.1 .3 -1.4
Fast /Medium 75 7.4 73.9 8.4 1.1 -1.0
Fast /High 75 11.1 73.7 11.4 1.3 -.3
Speed estimation error scores were computed by subtracting the subjects! estimate of the
average speed from the objectively calculated average speed of the original distribution
(presented in Table 4-2). An ANOVA was conducted on the absolute values of the speed
estimation error scores in order to determine if the size of error (regardless of direction)
differed as a function of the experimental variables of interest. The amount of speed
estimation error increased as variability increased (F(2,28) = 119.889, p < .001). All pair-
wise comparisons were significant (p < .001), suggesting that subjects made larger errors in
46
estimating speed in the high variability condition (M=3.5) than the medium variability
condition (M=2.4), and both the high and medium conditions were larger than the low
variability condition (M=1.2).
Figure 4-2. Actual and estimated mean speed as a function of speed and variability.
4.4.2 Estimated Speed Windows
An analysis was conducted to determine if the size of the subjects! estimated speed
windows differed as a function of the experimental conditions. Recall that subjects were
asked to provide the smallest window that they were 95% sure would contain the actual
average speed of a target vehicle that shared a speed profile with the 20 vehicles just
observed. This measure was intended to assess how well subjects were able to
differentiate among the three levels of variability and provide speed windows that reflected
this variability. The width of the subjects! estimated speed window differed as a function of
variability (F(2,28)=11.12, p < .001), shown by the dashed lines in Figure 4-3. The size of
the estimated speed window was smallest when variability was low (M=5.7), followed by
medium variability (M=8.7), and then by the high variability condition (M=12.0). Paired t-
47
tests revealed significant differences among all three levels of variability (p < .001). This
provides evidence that subjects! estimated speed windows were sensitive to the variability in
the stimuli and supports the use of 95% CI as a measure of uncertainty. In other words, this
finding is evidence that subjects can indeed create intervals that reflect the amount of
variability inherent in the stimuli. No other main effects or interactions were observed.
To determine if the subjects! estimated speed window size differed significantly from the
actual speed interval, the 95% CIs around the mean subjects! estimated speed window
widths were examined. As can be seen in Figure 4-3, there were only two conditions for
which the 95% CI did not contain the actual speed interval width – the slow speed/low
variability and moderate speed/low variability conditions. In both of these conditions, the
subjects! estimated speed windows were wider than the actual speed intervals. This shows
that, for the most part, the subjects were capable of developing calibrated windows for the
speed estimation task.
Figure 4-3. Estimated speed windows relative to actual speed interval.
4.4.3 Estimated TOA
Table 4-4 presents the actual mean TOA and TOA interval size, the subjects! estimated
mean TOA and TOA window size, and the difference between the actual and estimated
scores. A 3 x 3 x 6 repeated measures ANOVA was conducted to determine if the subjects!
estimated TOA differed significantly as a function of the experimental conditions. Again, this
was conducted to determine whether the participants could distinguish among the
48
experimental conditions (speed and variability), and to evaluate whether performance
differed across the six repetitions due to learning, fatigue, or scenario effects.
A significant speed x variability interaction (F(4,56) = 96.66, p < .001) provided evidence that
subjects! estimated TOAs differed as a function of the experimental independent variables.
As can be seen by the dashed lines in Figure 4-4, in the slow speed condition subjects
estimated longer TOAs in the high variability condition than for the medium condition (p <
.001) and both were higher than the low variability condition (p < .001 for both). In the
moderate speed condition, estimated TOAs in both the high and medium variability condition
were significantly higher than the low variability condition (p < .05), but there was no
difference between the medium and high variability conditions (p < .05). The same pattern
was observed for the fast speed condition – estimated TOAs in the high and medium
variability conditions were significantly higher than in the low variability condition (p < .05 for
both), but the high and medium variability conditions were not significantly different. These
results demonstrate that the subjects! TOA estimates were sensitive to changes in the
speed and variability of the stimuli.
Table 4-4. Actual TOA scores, estimated TOA scores & TOA estimation error scores
Actual TOAs Subjects! Estimated
TOAs
TOA Estimation Error (Actual –
Estimated)
Independent Variables (Speed / Variability) Mean
Interval Size
Mean Window
Size Mean
Window Size
Slow/Low 51.8 3.7 55.1 4.9 -3.3 -1.2
Slow/Medium 53.1 12.7 56.3 7.2 -3.2 5.5
Slow/High 56.2 25.2 59.5 11.2 -3.3 14.0
Moderate/Low 40.2 3.4 42.7 3.7 -2.5 -0.4
Moderate/Medium 40.8 7.1 43.7 5.2 -2.9 1.9
Moderate/High 41.9 12.3 44.4 6.9 -2.5 5.4
Fast /Low 24.1 1.2 24.6 2.7 -0.5 -1.5
Fast /Medium 24.2 2.4 25.1 3.5 -0.9 -1.1
Fast /High 24.4 3.8 25.3 4.1 -0.9 -0.4
49
Figure 4-4. Actual and Estimated TOA as a Function of Speed and Variability Graph shows 95% Confidence Intervals around the Estimated TOA.
Next, the subjects! estimated TOAs were compared to the actual mean TOAs by examining
the 95% CI around the mean of the subjects! estimated TOA to determine if the CI contained
the actual mean TOA (as shown in Figure 4-4). In conditions in which the CI does not
contain the actual TOA, it was concluded that the subjects! estimate was significantly
different than the actual TOA. In stark contrast to speed, the actual TOA was not contained
within any of the 95% CIs. This indicates that subjects! TOA estimates were significantly
different than the actual TOA for every condition. This suggests that subjects may perform
less well at the task of estimating TOA than they do at estimating speed. Subjects
overestimated TOA in all conditions.
TOA estimation error scores, given in Table 4-3, were computed by subtracting the subjects!
estimated TOA from the actual TOA of the population distribution. A repeated measures
ANOVA was conducted on the absolute values of the TOA estimation error scores to
50
determine if the amount of estimation error (regardless of direction) differed as a function of
the independent variables. The ANOVA yielded a speed x variability interaction (F(4, 56) =
3.941, p < .01). As seen in Figure 4-5, at slow speeds, absolute TOA estimation error was
greater in the high variability condition than the medium variability condition (p < .01) and
low variability condition (p < .05), but there was no difference between the low and medium
variability conditions (p > .05). At moderate speed, the same pattern was found – that is,
TOA estimation error was greater when variability was high than medium (p < .01) and low
(p < .05), but there was no difference between the low and medium variability conditions (p >
.05). (However, this last finding approached significance (p = .053).) At fast speeds, there
were no significant differences in the amount of TOA estimation error as a function of
variability.
Figure 4-5. Absolute TOA estimation error as a function of speed and variability.
(Absolute values are plotted, with +/- 1 SE).
4.4.4 Estimated TOA Windows
As shown in Figure 4-6, subjects! estimated TOA windows differed as a function of the
interaction between speed and variability (F(4,56) = 7.241, p < .001). Subjects! estimated
TOA windows were sensitive to the independent variables tested and reflected the relative
amount of variability (uncertainty) in the TOA of the stimuli. Of most interest was whether
51
subjects! TOA windows increased as a function of variability at each level of speed. Simple
effect tests revealed that this indeed was the case. Subjects! estimated TOA windows were
larger for the high variability condition than the medium variability condition, which in turn
were larger than the low variability condition (p < .05), for each level of speed.
Figure 4-6. Estimated TOA windows as a function of speed and variability. Graph shows 95% CIs around the estimated TOA window.
In order to determine whether the widths of the subjects! estimated TOA windows were
significantly different than actual TOA intervals, the 95% CIs for the three levels of variability
in each level of speed, shown in Figure 4-6, were examined to determine if they contained
the actual TOA interval width. In the slow speed condition, the width of the subjects!
estimated TOA window was significantly smaller than the actual TOA interval width in both
the medium and high variability conditions, but not the low variability conditions. Similarly, in
the moderate speed condition, the width of the subjects! estimated TOA window was again
significantly smaller than the actual TOA interval width in both the medium and high
variability conditions, but not the low variability conditions. In the fast speed condition,
however, the width of the subjects! estimated TOA window was significantly larger than the
actual TOA interval, in the low variability condition only.
A TOA window estimation error score (presented in Table 4-3) was calculated by subtracting
the width of the subjects! TOA window from the width of the actual TOA interval – in other
words, such that positive values represent overestimates of TOA window width. A 3 x 3 x 6
repeated measures ANOVA conducted on the absolute value of the TOA window estimation
Slow Moderate Fast
TO
A I
nte
rva
l W
idth
Low Medium High Low Medium High Low Medium High Variability Variability Variability
TO
A I
nte
rva
l W
idth
TO
A I
nte
rva
l W
idth
52
error score revealed a speed x variability interaction (F(4,56) = 98.503, p < .001), as seen in
Figure 4-7. Follow-up paired t-tests revealed that, at slow speeds, TOA window estimation
error was significantly higher when variability was high than medium (p < .001) or low (p <
.001) and error for the medium variability condition was significantly higher than low
variability condition, p < .001. At moderate speeds, the same pattern was observed, with
TOA window estimation error being significantly higher when variability was high than
medium (p < .001) or low (p < .001) and medium variability being significantly higher than
low variability (p < .001). However, for fast speeds, there was no significant difference in
TOA window estimation error as a function of variability.
Figure 4-7. TOA interval estimation error as a function of speed and variability (Absolute values, plotted with +/- 1 SE)
4.4.5 Estimated TOA Window Symmetry
In addition to the size of the subjects! estimated TOA window, the symmetry of the estimated
TOA window was also of interest. As was discussed previously, given a normal distribution
of speeds, the resulting TOA distribution is necessarily positively skewed – and furthermore
the amount of positive skew changes as a function of speed and variability – with greater
53
positive skew for stimuli with low speeds and high variability. To assess the degree to which
the subjects! estimated TOA window appropriately reflected this positive skew, a TOA
symmetry score was calculated for the actual distribution and for each subject!s TOA
window estimate, using the following equation:
TOA Symmetry Score = |UB –TOA| - |LB-TOA|;
where TOA = subjects! estimates of TOA,
LB = TOA Interval Lower Bound,
UB = TOA Interval Upper Bound (UB)
If the TOA symmetry score is equal to zero, this means that the window created by the
subject was perfectly symmetrical; if it is positive, then the window was positively skewed,
with the upper portion larger than the lower portion; and if negative, then the window was
negatively skewed with the lower portion larger than the upper portion.
Symmetry scores were first computed analytically from the underlying distribution for each
combination of speed and variability, as shown by the solid lines in Figure 4-8. The mean
TOA value was used as the measure of central tendency of TOA. As can be seen, the
actual TOA intervals were either symmetrical or positively skewed. The skew was highest in
the slow speed/high variability condition. This is consistent with the inverse relationship
between speed and TOA, as described previously.
Next, TOA symmetry scores were calculated for each subject!s estimated windows around
the subject!s own estimate of TOA. As can be seen by the dashed lines in Figure 4-8, the
subjects! symmetry scores also tended to be positive (overall mean = 0.2), therefore
suggesting that TOA windows provided by subjects were slightly positively skewed. Note
that for fast speeds the amount of skew was low and quite comparable to the actual
symmetry score; however, for slow speeds with high variability, symmetry scores were
positively skewed, but substantially less skewed than the actual symmetry scores.
54
Figure 4-8. Actual and estimated TOA symmetry scores. Estimated scores are plotted with 95% CIs.
The 95% CIs around the subjects! symmetry scores were examined to determine if they
contained the actual symmetry score. If the 95% CI did not contain the actual symmetry
score, the subjects! symmetry score was considered significantly different than the actual
symmetry score. This was the case for four speed/variability combinations: slow/medium,
slow/high, moderate/medium, and moderate/high. As can be seen in Figure 4-8, in these
four conditions for which the 95% CI did not contain the actual symmetry score, the actual
symmetry score was higher than the upper-bound of the subjects! 95% CI, reflecting that
subjects underestimated the amount of positive skew of the TOA interval. The graph shows
that subjects! estimated windows increased in positive skew as variability increased,
particularly in the slow condition, just as for the actual distributions; however, they
underestimated the magnitude of the skew.
Based on subjective comments made during the debrief session, most subjects detected
that the distribution was skewed. For example, many referred to "laggers! or "stragglers! –
the one or two or three vehicles that seemed substantially later than the rest of the sample.
However, most subjects suggested that they did not know how to factor these extreme
values into their interval estimations. Two strategies were prevalent. Some subjects took the
strategy of excluding the outliers altogether and considered them statistical anomalies.
Others chose to factor the outliers in by adjusting the window boundary "a little bit!, but again
expressed uncertainty as to how much to adjust the window. A hybrid strategy was also
55
implemented by other subjects, in that they would disregard a single outlier, but adjust the
interval boundary if there were 2 or 3 extremes.
4.5 DISCUSSION
4.5.1 Summary of Results
One objective of this research was to determine how well subjects estimated speed and
TOA. As expected, subjects! estimates of both speed and TOA differed among the levels of
speed, suggesting that the mean values chosen for the three distributions tested in this
experiment were perceived by the subjects as coming from different distributions, even for
the highly variable conditions. As expected, based on research of estimates of central
tendency, as summarized by Pitz (1980), subjects performed quite well at the task of
estimating mean speed, with only two conditions in which subjects significantly
underestimated speed as compared to the actual, or objectively computed, speed. In
estimating TOA, on the other hand, subjects over-estimated TOA compared to the actual
TOA for all conditions. This is a marked deviation from Pitz!s research, which would have
predicted that, when asked to estimate central tendency for other tasks, estimation error is
minimal.
It could be that TOA is inherently different from the types of stimuli used in other studies.
For example, it is likely that the positively skewed nature of the TOA distribution could have
increased TOA estimation error, as will be described later. Alternatively, it may be the case
that subjects weren!t attempting to estimate the mean of the distribution, but instead were
estimating some other measure of central tendency, such as the mode or median. During
the debrief, some subjects stated that they selected the time when the "middle group! of
vehicles finished the route – which supports the possibility that subjects were estimating the
median.
A second, and more important, goal was to determine how well subjects estimated the
amount of uncertainty associated with both speed and TOA. Estimated window size was
used as a measure of the subjects! estimate of uncertainty, in that larger windows reflected
larger estimates of uncertainty. For both speed and TOA, subjects! estimated window sizes
increased as a function of stimulus variability, providing support for the use of 95% intervals
56
as a measure of uncertainty. It was hypothesized that the size of the subjects! estimated
windows would be smaller than the actual intervals, since other subjects in the literature
tended to be overconfident in their estimates of uncertainty and not factor in extreme values
appropriately. This was not the case for speed windows. In fact, subjects produced well-
calibrated speed intervals and in only two conditions was there an indication of error in the
size of the estimated speed window – and in both cases the subjects overestimated, rather
than underestimated, the size of the window. TOA window estimation error, however,
depended on the speed of the stimuli. In general, for fast stimuli, subjects slightly
overestimated window size; however, for moderate and low speeds, subjects tended to
underestimate window size, as hypothesized. The amount of this underestimation was
greater for higher variability conditions than medium and low variability. The fact that
subjects did produce well-calibrated speed intervals also provides support for the
experimental technique. Any mis-calibration of TOA intervals can thus more likely be
attributed to real bias in TOA uncertainty estimation, as opposed to a general over- or under-
confidence bias in interval estimation due to the experimental technique.
Finally, it was hypothesized that subjects would create TOA windows that were symmetrical,
neglecting to account for the positive skew of the distribution. Although subjects! windows
were positively skewed, they still underestimated the amount of positive skew, particularly in
the slow/high variability condition.
4.5.2 Limitations of the Current Experiment and Suggestions for Future Studies
Being the first in a series of studies to explore how subjects estimate TOA and TOA
uncertainty, the stimuli and the task were quite simple. For example, the routes were
straight lines of equal length and all vehicles started at the same time. This simplicity may
have produced experimental artefacts that the subjects used in their estimates. During the
debrief, some subjects talked about "watching time! or using "visual patterns! of the vehicles
as they finished their routes. This allowed them to "see! the distribution of TOAs as it
unfolded. In essence then, subjects may have been simply describing the visible profile of
TOAs of the 20 vehicles, rather than applying their knowledge of speed variability to
estimate TOA. Future studies with added stimulus complexity, such as staggering the start
times or positions of the vehicles, should add to the robustness of these findings. In addition,
57
a different experimental paradigm in which subjects undergo a substantial training period to
learn to characterize speed profiles, which are later !applied" in experimental trials, may also
be more informative. This experimental paradigm will be used in the following experiment.
One particularly interesting finding in this experiment was how subjects factored in !extreme"
vehicles – that is those that are much faster or much slower than the others. Both the
objective data and the subjective comments made during the debrief session revealed that
subjects were uncertain how to treat extreme values and how to determine their effect on
TOA or TOA uncertainty. Indeed, this is likely the most difficult task associated with
estimating TOA and TOA uncertainty. With samples of 20 vehicles, there was usually only
one extreme vehicle, and many subjects (but not all) noted that they tended to ignore this
vehicle as a statistical anomaly. The strategy of dealing with extremes, however, was less
clear when there were two or three vehicles that were much slower or faster than the others.
Here, subjects expressed some uncertainty as to how to account for these vehicles.
Therefore, subsequent studies that show larger samples of vehicles to the subjects (i.e., 100
or more) may provide a more realistic opportunity to learn how subjects factor small
numbers of extremes into their estimation process. Having said this, however, it is believed
that 20 vehicles may be at or above the limit of the number of vehicles that a human subject
can process at one time. As such, the follow-on experiment will display a set of 100 vehicles
across five screens, rather than asking subjects to process 100 vehicles at the same time.
4.5.3 Implications for the Next Generation Aviation System
The airspace system is advancing to a !4-D" navigation environment, in which aircraft are
required to not only follow the conventional spatial routes, but also conform to a commanded
arrival time. Of course, for overall system efficiency, the notion of 4-D control has to be
applied to airport surface (taxi) operations as well, to ensure that aircraft are able to depart
on time and meet their next 4-D waypoint. (Note that current convention in the aviation
community utilizes the term “4-D navigation” also on the surface, even though altitude is
fixed.) On the surface, the goal is to minimize wait time by sequencing aircraft so they arrive
at each active intersection !just-in-time" to be able to cross it without waiting for other aircraft
to pass. In this environment, arriving early is as detrimental to system traffic flow as is
arriving late. For airport surface operations, this move to 4-D navigation will mean that pilots
will not only have to navigate the taxiways in order to get from gate to runway (and vice
58
versa), but also manipulate the aircraft speed (without the advantage of auto-pilot) so as not
to arrive too early or too late. Also, controllers must be able to estimate TOA and assign an
appropriate degree of uncertainty to the TOA estimate, based on the speed profile of the
aircraft and adjust traffic sequencing, or re-route traffic accordingly.
The results of this experiment, if they hold true after further, more robust, investigations,
suggest that, if an air traffic controller knows that a particular aircraft adheres to a certain
speed profile, such as slow and variable, or fast and consistent, they will tend to
overestimate the aircraft!s TOA – that is, a controller will estimate that the aircraft will arrive
later than it actually will – and more so for slower aircraft than faster aircraft. Furthermore,
for slow and moderate speed conditions, controllers will have a tendency to underestimate
uncertainty associated with the estimated TOA, which could result in sequencing aircraft too
close together, or not leaving a wide enough time window (or space envelope) around each
aircraft. Finally, the results of the symmetry analysis suggest that, particularly at slow and
moderate speeds with high variability, controllers will underestimate the likelihood that an
aircraft will arrive late to its destination.
59
CHAPTER 5: EXPERIMENT 3 – TOA UNCERTAINTY ESTIMATION AND
EXTRAPOLATION
5.1 INTRODUCTION
This experiment explored participants! ability to develop speed profile expectations by
learning a speed profile through training with feedback, and then to apply this speed profile
expectation to estimate TOA uncertainty. In addition, the ability to identify and apply the
appropriate speed profile based on observation of an individual vehicle was examined.
Thirdly, the ability to extrapolate TOA uncertainty estimates from known to unknown speed
profiles was examined.
5.1.1 Addressing Limitations of Experiment 2
This experiment addressed two limitations that were identified in experiment 2 (Chapter 4):
• Subjects appeared to describe the characteristics of TOA data displayed for each of
the 20 vehicles just observed, rather than apply their global knowledge of each
speed profile to estimate TOA.
• With only 20 vehicles per sample in experiment 2, subjects may not have had
enough data to properly understand and characterize the TOA distribution.
Separating the process of learning speed profiles in the training session from applying them
to estimate TOA uncertainty in the experimental trials was expected to minimize the
limitations of experiment 2, in which subjects were simply describing the TOA distribution of
20 vehicles at a time. In the present experiment, subjects were exposed to a training
program in which they learned speed profiles by exposure to a larger number of vehicles
(100 vehicles per speed profile) than in experiment 2. After the training session, subjects
were asked to apply their knowledge of the speed profiles by estimating TOA uncertainty for
subsequent vehicles – which either followed a trained speed profile or a new speed profile.
A third change in the experimental protocol was made in an attempt to improve the subjects!
understanding of the task by clarifying the concept of the intervals. In the previous
experiment, subjects were asked to provide “the smallest TOA window that they were 95%
sure would contain the vehicle!s actual TOA.” In this follow-on experiment, these
instructions were augmented to better define “95% sure”. Specifically, subjects were
instructed to select a window that would capture 95 of 100 vehicles. Because subjects saw
60
100 vehicles of each speed profile it was expected that this would make the task more
concrete and meaningful.
5.2 RESEARCH QUESTIONS
This experiment set out to address 5 research questions as listed below, and shown
schematically in Figure 5-1, in relation to the model of TOA Uncertainty developed in
Chapter 2. These research questions were addressed across three phases of a single
within-subjects experiment. The experiment phase is listed in parentheses after each
question.
1. Can subjects develop an internal model of a speed profile based on the average speed and speed variability? (Phase 3a)
2. Can subjects apply their internal model of a speed profile to estimate TOA uncertainty – and does this differ as a function of average speed or speed variability? (Phase 3b)
3. Can subjects extrapolate from a trained distance to estimate TOA uncertainty for a new, untrained distance? (Phase 3b)
4. Based on a period of observation of a single vehicle, can subjects identify the vehicle!s speed profile from their repertoire of known speed profiles and apply knowledge of this profile to accurately estimate TOA uncertainty? (Phase 3c)
5. Can subjects extrapolate from trained speed profiles to accurately estimate TOA uncertainty for route distances and speed profiles on which they were not trained. (Phase 3c)
Figure 5-1. Research questions mapped to the preliminary model of TOA uncertainty (Numbers in parentheses refer to phases of the experiment)
61
5.3 METHOD
5.3.1 Subjects
Sixteen subjects were recruited from colleges and the community local to NASA Ames
Research Center, where the experiment was conducted. Subjects were pre-screened for
colour blindness and instructed to wear glasses/contact lenses normally used for computer
work. Subjects! ages ranged from 21 to 54, with a mean age of 35. The range of education
levels possessed by the participants included high school (3 participants), two-year college
diploma (2 participants), four-year university (7 participants), and graduate school (4
participants). Seven of the subjects had taken at least one university-level course in
statistics. Prior to participation, all subjects successfully completed a vision test to ensure
that they could read text in the same font style, size, and colour used for the experimental
stimuli under lighting conditions that were identical to that of the actual experimental trials.
The experiment lasted approximately 3.5 hours and subjects were paid $10 per hour for
their participation.
5.3.2 Experimental Design
This experiment consisted of three phases, as described below. All subjects completed
each of the three phases in a single session, and in the same order. Prior to embarking on
the first phase, subjects first read a set of instructions (see Appendix C1) that outlined the
purpose and tasks required for each phase, signed an informed consent form, and
completed a short demographic questionnaire (see Appendix C2). Upon completion of the
third phase, subjects participated in a post-study debriefing session consisting of a semi-
structured interview (see Appendix C3).
5.3.2.1 Phase 3a: Developing a Library of Speed Profiles
The goal of this first phase of the experiment was to train subjects to accurately characterize
four different speed profiles through exposure to a large number of vehicles and by providing
feedback.
Task Description. As shown in Figure 5-2, subjects observed on a single screen 20 vehicle
icons that shared a common speed profile (i.e., all speeds were randomly drawn from the
same normal distribution), travelling simultaneously along the same length route (medium
62
length: 500 pixels). Throughout this phase, all vehicles that shared the same speed profile
also shared the same colour-coded icon. Each route was divided into 10 segments with
visible vertical hatch marks, and the speed changed for each segment. This was similar to
the stimuli used in experiment 2, except the segment hatch marks were not shown in
experiment 2 and the elapsed time readout shown in experiment 2 was not shown in this
current experiment.
Figure 5-2. Phase 3a Stimuli. Each red dot represents a vehicle. Each three-digit label (e.g., 36.4) represents current ground speed. The twenty routes were identified sequentially, with 1 at the top and 20 at the bottom.
63
After the last vehicle had completed the route, subjects were asked to report the number of
the vehicle that arrived first and last. (This task was aimed at ensuring that subjects actually
attended to the distribution of arrival times.) Next, subjects were asked to characterize the
speed profile by estimating the average speed of the profile, and providing a lower bound
and upper bound for the smallest range of speeds that would capture 95% of the individual
average speeds, in the same manner as experiment 2. Subjects were given feedback after
every trial that indicated whether their answers were “Correct”, “Too High” or “Too Low”
relative to the mean speed and 95% CIs of the population distribution. The feedback was
provided on the subject!s estimate of average speed as well as both the lower bound and
the upper bound individually. The feedback algorithm had a buffer of 1 unit, such that, if a
subject!s answer was within plus or minus 1 of the actual answer, the algorithm reported
“Correct”. In total, subjects experienced five consecutive trials of each colour-coded speed
profile – for a total of 100 vehicles per speed profile. Upon completion of the fifth trial for
each speed profile, subjects also received the actual mean, lower bound and upper bound,
as computed objectively from the speed profile distributions. Subjects then answered the
three questions below pertaining to the speed profile, using their full knowledge of all of the
vehicles that shared that profile – that is, all 100 vehicles observed.
Q1. Based on their average speed, are the vehicles within this profile more likely travelling on a six-lane freeway (i.e., fast) or a two-lane rural road (i.e., slow)? Please circle one:
1 2 3 4 5 6 7 Definitely Definitely two-lane rural six-lane freeway
Q2. Based on their speed fluctuations, are the vehicles within this profile more likely
travelling with cruise control ON or OFF? Please circle one: 1 2 3 4 5 6 7 Definitely Definitely cruise control ON cruise control OFF
Q3. Please characterize the time of arrival of vehicles that share this speed profile.
(use words, symbols, numbers, or pictures).
The purpose of the first two questions was to assess how subjects categorized each profile
on the two dimensions: speed (low or high) and variability (low or high) and to determine if
64
their categorization matched the stimuli. With regards to Question 3, subjects were
encouraged to make notes to characterize the TOA distribution using words, symbols,
numbers, or pictures. For example, subjects could note that the TOAs were !spread out" or
!clumped together", estimate a number of seconds from the first to the last vehicle, or use
symbols or pictures to depict the spread graphically. They were instructed that these notes
would be important for completing Phase 2 and 3 of the experiment, in which they would be
required to estimate TOA, and that these were the only notes that they could carry through
to the subsequent experimental phases. These notes also served to emphasize the
importance of utilizing their knowledge of the speed profile of the full set of 100 vehicles, not
just of a single vehicle.
Subjects completed this task for four different speed profiles made up of speed/variability
pairings adopted from Experiment 2: slow/low, slow/high, fast/low, fast/high, summarised in
Table 5-1. Each speed profile was labelled by colour-coded icons, allowing subjects to
differentiate among the four speed profiles – slow/low = red; slow/high = yellow; fast/low =
green; fast/high = blue. The five trials for each speed profile were blocked, and the order in
which the speed profile blocks was presented to each subject was determined using a Latin
Square design.
Table 5-1. Phase 3a Experimental Design
Speed Profile
(Speed/Variability)
Profile
Labelled
Number
of Trials
Number of
Vehicles per
Trial
Route
Distances
(in pixels)
Slow/Low Yes 5 20 Medium (500)
Slow/High Yes 5 20 Medium (500)
Fast/Low Yes 5 20 Medium (500)
Fast/High Yes 5 20 Medium (500)
5.3.2.2. Phase 3b: Estimating TOA Uncertainty for Trained Profiles
The goals of this phase were to:
1) Assess whether subjects could apply their knowledge of the four learned speed
profiles to accurately estimate TOA uncertainty; and
65
2) Assess subjects! ability to extrapolate to distances other than that distance (medium)
for which they were trained.
Task Description. In this block of trials, subjects were shown a dynamic simulation of a
single vehicle with a digital speed readout as it traversed a medium length route (500 pixels,
as used in Phase 3a; see Figure 5-3 top). All vehicles were colour coded to match one of
the four speed profiles learned in Phase 3a. Subjects were instructed that this was a
prototypical vehicle that was representative of all of the vehicles that shared that profile, and
when answering the questions they should consider all vehicles that shared that same
colour-coded speed profile. The stimuli in this phase, shown in Figure 5-3, were the same
as in Phase 3a, except that only one vehicle was shown at a time. The speed readout was
displayed, but the elapsed time was not displayed. The speed profile always matched one of
the four trained profiles, and was identified using the same colour-coding scheme used in
Phase 3a. The vehicle!s speed changed with each segment of the route, and speeds were
drawn from the normal distribution that characterized the speed profile.
Figure 5-3. Phase 3b experimental stimuli. The top panel shows the dynamic simulation of one vehicle traversing a medium-length route adhering to the "red! speed profile (Slow/Low Variability). The icon changes speed at each segment (vertical hatch mark). The bottom panel shows the static probe screen presented after the dynamic simulation (top) was completed, with the same speed profile (red colour-code) and a visual representation of the route!s distance remaining (in this case a short, 250 pixel route). The top dynamic simulation was shown first, followed by the bottom static screen.
Static Probe Screen
Dynamic Simulation
66
After the vehicle had completed 10 speed change segments and reached the end of the
route, the vehicle looped to the beginning of the route (see Figure 5-3, bottom) and its
movement was paused with a new distance remaining that was either short (250 pixels),
medium (500 pixels), or long (1000 pixels). An !expected TOA" was presented to the
subjects that reflected the time (in seconds) that the vehicle was expected to complete the
route. (The time shown was the mean of the TOA distribution for the distance remaining.)
Subjects were asked to produce TOA intervals that reflected the smallest window that they
were 95% sure would contain the actual TOA, in the same manner as experiment 2. It was
emphasized in the written and on-screen instructions that the subjects should aim to
produce intervals that capture 95 of 100 vehicles in the profile.
As shown in Table 5-2, stimuli consisted of four speed profile conditions at each of three
distances (250, 500, and 1000 pixels). Recall that routes in the training block were 500
pixels long – thereby requiring subjects to apply their knowledge of speed to estimate TOA
windows for the same distance (500 pixels) and also to extrapolate to a shorter route (250
pixels) or a longer route (1000 pixels).
Subjects completed two familiarization trials with the experimenter present: the slow/high-
short route and the fast/low–long route. Then subjects completed one block of 12 training
trials (4 speed profiles x 3 distances), followed by four blocks of 12 experimental trials. The
presentation order of the twelve trials within each block was determined randomly.
Table 5-2. Phase 3b experimental design
Speed Profile (Speed/Variability)
Profile Trained
Profile Labelled
Number of Trials Route Distance
(pixels)
Slow/Low Yes Yes 1 training +
4 experimental 250, 500, 1000
Slow/High Yes Yes 1 training +
4 experimental 250, 500, 1000
Fast/Low Yes Yes 1 training +
4 experimental 250, 500, 1000
Fast/High Yes Yes 1 training +
4 experimental 250, 500, 1000
67
Hypotheses. If the findings from Experiment 2 hold, it was expected that subjects would
overestimate TOA uncertainty in the fast conditions, particularly the low variability condition
(that is, they were expected to produce TOA windows that were larger than the actual TOA
intervals). For slow speeds, it was expected that estimated TOA windows would be smaller
than actual TOA intervals, and more so for the high variability condition than low variability
condition. The magnitude of TOA estimation error was expected to be smaller than for
experiment 2 because the increased training protocol should yield improved estimates.
Further, if subjects were able to extrapolate from the trained route distance (500 pixels) to
new, untrained distances, a main effect of route distance would be expected, with smaller
estimated TOA windows for shorter distances, and longer TOA windows for longer
distances. Absence of a significant main effect, assuming sufficient power, would suggest
that subjects could not extrapolate from trained distances to untrained distances.
5.3.2.3. Phase 3c: Estimating TOA Uncertainty for New, Untrained Profiles
The goal of this phase was to determine if subjects could identify a known speed profile
based on a short period of observation and apply knowledge of this profile to estimate TOA
uncertainty. A second goal was to assess how well subjects could extrapolate from the
trained speed profiles to assess TOA uncertainty for speed profiles on which they were not
trained.
Task Description. These experimental trials were identical to the trials in Phase 3b, with
three exceptions:
1) The speed profile was not labelled (through colour coding) in any trials;
2) Some trials matched the four trained speed profiles, but some did not, as shown in Table
5-3;
3) All route distances were medium length (500 pixels, the same as the trained route
distance).
This enabled an evaluation of the subjects! ability to extrapolate from trained profiles to new
untrained profiles, presented in Table 5-3 as shaded rows. Subjects completed three trials
for each speed profile. The nine speed profiles were presented in random order in three
blocks. Subjects were informed that not all vehicles shared one of the four trained speed
profiles.
68
Prior to estimating the TOA windows, subjects were asked to 1) determine if the speed
profile was one of the trained profiles; 2) identify the speed profile (by colour) that it most
closely resembled (or state the colour of the trained profile they believed it matched), and 3)
rate the similarity of the vehicle!s profile to the selected speed profile on a scale from 1 (very
low) to 5 (very high).
Table 5-3. Phase 3c experimental design. (Shaded rows are new, untrained, speed profiles)
Hypotheses. For trained profiles, it was expected that subjects would overestimate TOA
uncertainty in the fast conditions (that is, estimate TOA windows that are larger than the
actual TOA intervals) and underestimate TOA uncertainty (estimate TOA intervals that are
smaller than actual TOA intervals) in the slow conditions, and more so for the high variability
condition than low variability condition. TOA estimation errors that are larger for Phase 3c
trials than 3b trials would indicate that subjects were not able to identify the speed profiles
solely from the observed stimuli.
Speed Profile
(Speed/Variability)
Profile
Trained
Profile
Labelled
Number of
Trials
Route
Distance
(in pixels)
Slow/Low Yes No 3 500
Slow/Medium No No 3 500
Slow/High Yes No 3 500
Moderate/Low No No 3 500
Moderate/Medium No No 3 500
Moderate/High No No 3 500
Fast/Low No No 3 500
Fast/Medium Yes No 3 500
Fast/High Yes No 3 500
69
5.4 RESULTS
5.4.1 Phase 3a: Developing a Library of Speed Profiles
Recall that the goal of phase 3a was to expose subjects to 100 vehicles travelling with a set
of shared speed profiles to enable them to develop a mental model of the profile in terms of
average speed and speed variability. Analyses were conducted to evaluate how well
subjects were able to estimate the mean speed and speed windows for each speed profile.
Table 5-4 shows actual and estimated mean speed and speed windows for the fifth (final)
trial of each speed profile, and a difference score representing actual minus estimated
values.
Table 5-4. Actual speed scores, estimated speed scores, and speed estimation error scores
Actual Subjects! Estimates Estimation error
(Actual – Estimate) Speed
Profile Mean Interval Mean Window Mean Window
Slow/Low 35 3.75 34.7 4.13 0.3 -0.38
Slow/High 35 11.15 34.7 9.76 0.3 1.39
Fast/Low 75 3.75 75.5 4.26 -0.5 -0.51
Fast/High 75 11.15 73.8 9.38 1.2 1.77
*Data shown in table are for last trial of each speed profile
5.4.1.1 Estimated Mean Speed
The first element required to develop a model of the speed profile was to estimate mean
speed. Figure 5-4 shows the subjects! estimated mean speed (dashed lines) on the fifth
(final) trial for each speed profile, as compared to the actual mean speed (solid lines). A 2 x
2 x 5 ANOVA revealed no significant three-way interaction. But, subjects were sensitive to
the difference in speed, estimating the mean speed for fast stimuli (M = 74.7) as faster than
for slow stimuli (M=34.7) (F(1,15)=5630.17, p<.001). The difference between subjects!
estimated mean speed and the actual mean speed was negligible (p>.05) in all conditions.
The subjects! estimates of mean speed did not differ as a function of variability
(F(1,15)=1.124, p>.05). Although not shown in Figure 5-4, mean speed estimates did differ
70
as a function of trial (F(4,60)=3.212, p<.05), with subjects underestimating mean speed on
the first trial, but quickly calibrating toward the actual mean for the remaining trials.
Figure 5-4. Estimated mean speed on 5th and final trial (shown with 95% CI)
5.4.1.2 Estimated Speed Windows
The second element required to develop an internal model of a speed profile was the
subjects! ability to estimate the speed windows (or the window that captures 95% of the
mean speeds). A 2 x 2 x 5 ANOVA was conducted. Neither a three-way interaction nor a
main effect of average speed on speed window estimation was found (p>.05). This
supports the premise that the subjects understood that actual speed windows do not differ
as a function of speed, but only as a function of variability, as can be seen in Table 5-4. A
main effect of variability (F(1,15) = 47.63, p<.001), on the other hand, revealed that subjects
did estimate larger speed windows for the high variability condition (M = 8.85) than for the
low variability condition (M = 4.82), reflecting an understanding of the relationship between
interval size and variability. A significant trial x variability interaction (F (4,60)=12.61,
p<.001) revealed that estimated window sizes increased as a function of trial repetition in the
high variability condition, but decreased in the low variability condition. Similarly, a Trial x
Speed interaction (F (4,60)=2.87, p<.05) revealed that subjects! window size estimates
increased as trials progressed for fast speeds and decreased for slow speeds. These
71
interactions reflect an improvement in performance due to feedback, as will be shown in the
following section.
Figure 5-5 presents the speed window difference scores (actual minus estimated speed
window size) as a function of speed, and trial, where a perfectly calibrated speed window
estimate yields a difference score of zero. A 2 x 2 x 5 ANOVA revealed a significant speed x
trial interaction (F(4,60)=2.874, p<.05). (No significant three way interaction was found.) As
can be seen in Figure 5-5, initially subjects tended to overestimate the size of the speed
windows for the slow speed trials and underestimate the speed windows for the fast speed
trials – however, with feedback after each trial, both converged towards the actual speed
interval size, with a low difference score by the fifth trial. Analyses of just the fifth trial
revealed no significant difference in speed window difference scores as a function of speed
(p>.05).
Figure 5-5. Speed window difference scores as a function of speed (slow, fast) and trial number (shown plotted with ±1 Std Error)
A variability x trial interaction was also observed (F(4,60)=12.607, p<.001), as seen below in
Figure 5-6. Subjects overestimated speed windows for low variability stimuli and
underestimated speed windows for high variability stimuli. A subsequent analysis of the final
trial showed that the speed window difference score was significantly higher for low
variability conditions than high variability conditions (F(1,15)=9.218, p<.05).
72
Figure 5-6. Speed window population difference scores as a function of variability (low, high) and trial (shown plotted with ± 1 std error)
In summary, in a small number of trials with performance feedback, subjects were able to
effectively develop an internal model of each speed profile and characterize both mean and
speed intervals with little error.
5.4.2 Phase 3b: Estimating TOA uncertainty for trained profiles
Recall that the goal of this second phase of the experiment was to determine if subjects
could apply their internal model of the four speed profiles to accurately estimate TOA
uncertainty for trained distances. A second goal was to determine if subjects could use their
knowledge of trained speed profiles to estimate new (untrained) distances. Each subject
completed five trials for each speed profile x distance combination, in a random order. The
first trial of each condition was considered practice and the analyses that follow were
conducted on the remaining four trials.
5.4.2.1. Estimated TOA Uncertainty
Table 5-5 presents the actual TOA interval size, the subjects! estimated TOA window size,
and the difference between the actual and estimated window size. Recall that the expected
TOA was provided to the subjects to eliminate potential confounds associated with
misestimating the TOA itself.
Trial
73
Table 5-5. Actual TOA scores, estimated TOA scores, and TOA estimation error scores
Independent Variables
Speed Variability Distance
Actual TOA Interval
Estimated TOA Window
TOA Estimation Error (Actual –
Estimated)
Slow Low Short 2.8 4.6 -1.8
Slow Low Medium 5.6 5.5 0.1
Slow Low Long 11.2 6.6 4.6
Slow High Short 12.5 9.3 3.2
Slow High Medium 25.0 10.3 14.7
Slow High Long 50.0 12.4 37.6
Fast Low Short 0.6 2.7 -2.1
Fast Low Medium 1.2 3.4 -2.2
Fast Low Long 2.4 3.8 -1.4
Fast High Short 1.9 4.6 -2.7
Fast High Medium 3.8 5.4 -1.6
Fast High Long 7.6 6.6 -1.0
A 2 x 2 x 3 x 4 repeated measures ANOVA was conducted to determine if the subjects!
estimated TOA window differed as a function of the experimental conditions (speed,
variability, distance, repetition). Again, this was conducted to investigate whether the
subjects could distinguish among the experimental conditions, and to evaluate whether
performance differed across the four repetitions due to learning, fatigue, or scenario
differences. The analysis revealed a significant main effect of distance (F(2,30)=13.979,
p<.001), and post-hoc comparisons revealed that subjects! estimated TOA window size
increased as a function of distance. TOA windows were smallest for short routes (M=5.29),
followed by medium routes (M=6.15), and long routes (M=7.34). All pairwise comparisons
were significantly different (p<.01). This suggests that subjects did factor the distance of the
route into their TOA uncertainty estimation.
A significant speed x variability interaction (F(1,15)=8.95, p=.009) provided evidence that
subjects! estimated TOA uncertainty differed as a function of the two main experimental
independent variables. As can be seen by the dashed lines in Figure 5-7, at slow speeds
74
the difference in estimated window size between low (M=5.55) and high (M=10.66)
variability was greater than the difference between low (M=3.3) and high (M=5.5) variability
at fast speeds.
Figure 5-7. Estimated TOA window size as a function of speed (slow, fast) and
variability (low, high). Error bars represent ±1 std error
Next, the subjects! estimated TOA window was compared to the actual TOA interval by
examining the 95% CI around the subjects! estimated TOA window size, to determine if the
actual TOA window size was contained within the 95% CI. Figure 5-8 shows the subjects!
estimated TOA window size (dashed lines) and the actual TOA interval size (solid lines) as a
function of distance for each combination of speed and variability. As indicated by the circles
in the figure, in all but two conditions (slow/low variability/medium distance and fast/high
variability/long distance) the 95% CI around the subjects! estimated window size did not
contain the actual TOA interval size, indicating that the subjects! estimate was significantly
different than the actual interval size. As can be seen in the upper left graph in Figure 5-8,
for the slow/low variability condition, the subjects overestimated TOA window size for short
distances and underestimated TOA window size for long distances. For the slow/high
variability condition (upper right), the subjects underestimated TOA window size for all
distances. The opposite was true for the fast/low variability condition (lower left) and
75
fast/high variability conditions, in which the subjects overestimated TOA window size for all
distances (with the exception of fast/high variability/long distance).
Figure 5-8. Actual (solid lines) and estimated (dashed lines) TOA windows as a function of speed (slow, fast), variability (low, high), and distance (short, medium, long). Graph shows 95% CIs around the estimated TOA window. Circles show data where actual interval widths fall outside of the 95% CI of the subjects! estimates.
The pattern of overestimated TOA window size for short distances and underestimated TOA
window size for long distances can be seen more readily in Figure 5-9, which adjusts the
subjects! estimates to correct for the initial estimation error seen in the trained (medium)
distance. That is, when the subjects! TOA uncertainty estimate for the medium distance is
shifted to match the actual TOA uncertainty, one can clearly see that the subjects
systematically overestimate TOA uncertainty for short distances, and underestimate TOA
uncertainty for long distances.
Related to this finding, in the debrief session, all subjects were shown three scenarios of a
vehicle with the same speed profile travelling routes of different lengths (see Appendix C3),
to assess their understanding of the effect of distance on TOA uncertainty. In the first
scenario, the route distance was 4 miles, and the TOA window around an expected TOA of
320 seconds was 80 seconds. In the second scenario, subjects were asked to estimate the
TOA window for a vehicle with the same profile travelling a route that was half as long – 2
miles. The third scenario asked subjects to estimate the TOA window for an 8 mile route.
For the second (2 mile route) scenario, all but two subjects correctly estimated a TOA
76
window of 40 seconds (which was half the TOA window provided in scenario 1). One
subject estimated a shorter window (20 seconds) and one estimated a longer window (60
seconds). For scenario 3 (8 mile route, which was twice the distance of scenario 1), all but
four subjects correctly estimated the TOA window to be 160 seconds (or twice as long as
provided in Scenario 1). One subject estimated a shorter TOA window (90 seconds), while
three estimated longer TOA windows (180, 200, 320 seconds). While there were some
exceptions, the majority of the subjects demonstrated an accurate understanding of the
relationship between TOA uncertainty and route distance. Yet in the experiment, the
subjects systematically misestimated TOA uncertainty as a function of distance.
Figure 5-9. Actual (solid lines) and estimated (dashed lines) TOA windows as a function of speed (slow, fast), variability (low, high), and distance (short, medium, long), corrected for
initial estimation error at the trained (medium) distance.
5.4.2.2. TOA Uncertainty Estimation Error
TOA estimation error scores, shown in Table 5-5, were computed by subtracting the
subjects! estimated TOA window size from the actual TOA interval size generated from the
population distribution. A TOA window estimation error score of zero reflects a perfectly
calibrated TOA window estimate. A repeated measures ANOVA was conducted on the
absolute values of the TOA window estimation error scores to determine if the amount of
estimation error (regardless of direction) differed as a function of the independent variables.
As evident in Figure 5-10, the ANOVA yielded a speed x variability x distance interaction
(F(2,30) = 202.37, p<.01). Subsequent simple effect tests revealed a significant variability x
77
distance interaction for the slow speed condition (F(2,30)=258.364, p<.001). In the slow/low
variability condition, TOA window estimation error was not significantly different for short
(M=2.02) and medium (M=1.68) distances, p>.05), but TOA window estimation error was
larger for long distances (M=5.18) than both medium and short distances, (p<.001). In the
slow/high variability condition, TOA window estimation error was highest for long distances
(M=37.99), followed by medium distances (M=14.87) and short distances (M=5.27). All
pairwise comparisons were significantly different (p<.001). For fast speeds, no interaction
was found for variability x distance (p>.05). That TOA uncertainty estimation error was
greatest in the slow/high profile for long distances suggests that subjects treated this
condition as a symmetrical distribution, and failed to recognize the substantial increase in
TOA interval size associated with this condition.
Figure 5-10. TOA window estimation error (absolute values), as a function of speed (slow,
fast) and variability (low, high). (Plotted with ±1 SE).
5.4.2.3 Estimated TOA Interval Symmetry
In addition to the size of the subjects! estimated TOA window, the symmetry of the estimated
TOA window was also of interest. As was discussed previously, given a normal distribution
of speeds, the resulting objectively determined distribution of TOAs is necessarily positively
skewed – and furthermore the amount of positive skew changes as a function of speed and
variability – with the greater amount of positive skew for stimuli with slow speeds and high
78
variability. To assess the degree to which the subjects! estimated TOA windows
appropriately reflected this positive skew, a TOA symmetry score was calculated using the
following equation (and also used in experiment 2):
TOA Symmetry Score = |UB –TOA| - |LB-TOA|;
where TOA = subjects! estimates of TOA,
LB = TOA Interval Lower Bound,
UB = TOA Interval Upper Bound (UB)
If the TOA symmetry score is equal to zero, the interval created by the subject was perfectly
symmetrical; if positive, then the window was positively skewed with the upper portion larger
than the lower portion; and if negative, then the window was negatively skewed with the
lower portion larger than the upper portion.
Symmetry scores were first computed on the population data for each combination of speed
and variability as shown in Figure 5-11. As can be seen from the solid lines, the intervals
were either symmetrical or positively skewed. The skew was highest in the slow speed/high
variability condition. This is consistent with the inverse relationship between speed and
TOA, as discussed previously. Next, TOA symmetry scores were calculated for each
subject!s estimated TOA windows. As can be seen from the dashed lines in Figure 5-11, the
subjects! symmetry scores also tended to be neutral or slightly positive (Overall Mean = .05),
therefore suggesting that subjects provided TOA windows that were slightly positively
skewed. Note that for fast speeds the amount of skew was low and quite comparable to the
actual symmetry score; however, for slow speeds with high variability, symmetry scores
were substantially less skewed than the actual symmetry scores.
The 95% CIs around the subjects! symmetry scores were examined to determine if the CI
contained the actual symmetry score. If the 95% CI did not contain the actual symmetry
score, the subjects symmetry score was considered significantly different than the actual
symmetry score. This was the case for all combinations of variability and distance for the
slow condition, in which the actual TOA windows were more positively skewed than the
subjects! estimates, with the exception of slow/low variability/short distance, which was not
different than the actual symmetry score. In contrast, subjects! estimates of TOA symmetry
for the fast stimuli were not significantly different than the actual symmetry score for any
variability or distance combination.
79
Figure 5-11. Actual and estimated TOA symmetry scores. Estimated scores are plotted with 95% CIs. Top graph: slow speed; bottom graph: fast speed.
5.4.3 Phase 3c: Estimating TOA Uncertainty for Untrained Profiles
The goal of this third phase was to determine if, after observing an unlabelled vehicle
travelling a medium length route with 10 speed changes, subjects could determine whether it
was a speed profile for which they had been trained – and apply TOA windows accordingly.
A second goal was to determine whether subjects could extrapolate from the trained speed
profiles to assess TOA uncertainty for speed profiles on which they were not trained. In
contrast to Phase 3b, only one distance (medium route, 500 pixels) was used for all trials in
this phase. Recall that after viewing the vehicles, subjects were asked to first determine
80
whether the vehicle!s speed profile was new or old (i.e., one of the four they had learned in
training). Subjects were then asked to select which of the four trained profiles it was most
similar to – or state the profile they believed it matched – and provide a similarity rating from
1 (very low) to 5 (very high). Then, subjects estimated the TOA windows and entered them
in the same manner as Phase 3b.
5.4.3.1 Identification of Trained and Untrained Profiles
Subjects correctly identified 63% of the profiles as trained or untrained, with a mean of 17
correct identifications out of 27 trials (with a range of 12 – 21). As shown in Figure 5-12, the
correct identification rate for trained profiles (76%) was higher than for new, untrained
profiles (52%) (X2 = 26.27, p<.05).
Figure 5-12. Identification of "trained! and "untrained! profiles
5.4.3.2 Correct Identification of Speed Profile
Next, subjects were asked to identify which of the original four speed profiles the vehicle
was most similar to – or, if the subject stated that it was a trained profile, they were asked to
identify which one of the four profiles it was. Figure 5-13 shows the percent of trials that
were identified correctly for each of the four trained profiles. (Note that this analysis uses
only the four trained profiles, since there was no "correct! closest profile for the untrained
profiles). Identification rates were highest for the slow/low (88%), followed by slow/high
(73%), fast/high (67%) and fast/low (65%) profile. Chi square residual analyses revealed
Trained Untrained Speed Profile
Pe
rce
nt
Co
rre
ct
81
that the identification rate for the slow/low condition was above the mean percent correct
value and the fast/low and fast/high rates were below the mean percent correct value (X2=
7.8, p<.05). The higher identification rate for the slow conditions is likely attributed to the
skewed distribution yielding a range of TOAs that was much larger, and thus arguably more
salient, than for all other conditions.
Figure 5-13. Correct Identification of Speed Profile
5.4.3.3 Similarity Ratings
As summarised in Table 5-6, when asked to rate the similarity of the observed profile to the
trained profile that the subjects identified as being the closest, subjects rated the trained
profiles higher (more similar) than the untrained profiles. That is, when subjects reported that
it was one of the four trained profiles and it actually was one of the four trained profiles,
subjects rated the similarity rating higher (M = 4.2) than if subjects reported that it was a new
profile that they had not seen before and it was indeed a new profile (M = 2.6)
Table 5-6. Mean similarity ratings
Actual profile was: Subject Response
Trained Untrained
Trained 4.2 3.8
Untrained 3.0 2.6
Pe
rce
nt
Co
rre
ct
Slow/Low Slow/High Fast/Low Fast/High Speed Profile
82
5.4.3.4 Estimated TOA Window
Next, subjects were asked to estimate the TOA uncertainty window in the same manner as
in Phase 3b. Figure 5-14 shows the subjects! estimated TOA windows (dashed lines)
plotted with 95% CIs, and the actual TOA intervals (solid lines). A 3 x 3 x 3 ANOVA was
conducted using all trials (trained and untrained speed profiles) to determine if the subjects!
estimated TOA window size differed as a function of speed, variability, and trial. (Recall that
for this phase, a new speed (moderate/45) and a new level of variability (medium/6) were
introduced, creating three levels of each variable. Distance was held constant at medium, as
used for Phase 3a.)
Figure 5-14. Estimated and Actual TOA windows for both trained and untrained profiles
Results revealed a significant main effect of speed (F(2,30) = 17.24, p<.05). (No three-way
or two-way interactions were found.) Follow-up post-hoc tests did not reveal any significant
difference between the subjects! estimated TOA window for the slow speed condition
(M=6.7) and the estimated TOA window for the newly added moderate speed condition
(M=7.0), but it did reveal that both were significantly larger than the TOA window for high
speeds (M=4.3) (p<.001). This result suggests that subjects may have simply applied the
TOA window for slow speeds to the new moderate speed condition, rather than interpolating
between the slow and fast conditions.
83
Subjects! estimated TOA windows differed as a function of variability (F(2,30)=10.84,
p<.001). TOA window estimates for the low variability condition (M=4.8) were lower than for
the new medium variability condition (M=6.353) (p<.05), and for the high variability condition
(M=6.9) (p<.05). Subjects' TOA window estimates for medium and high variability were not
significantly different (p>.05). This result suggests that subjects may have applied the TOA
window for high variability to the new medium variability condition.
There was a significant effect of repetition (F(2,30)=4.33, p<.05). Results revealed that
estimated TOA windows were shorter on Trial 1 (M=5.7) than Trial 2 (M=6.4) (p<.05), but
Trial 3 (M=6.0) was not significantly different than either Trial 1 or 2 (p >.05).
Next, subjects! estimated TOA windows (dashed lines in Figure 5-14) were compared to the
actual TOA intervals (solid lines in Figure 5-14). In the slow speed condition, subjects
underestimated the TOA window size for medium and high variability conditions. In the
moderate speed condition, subjects overestimated the TOA window for low variability and
underestimated the TOA window for the high variability condition. In the fast speed
condition, subjects overestimated the TOA window for both low and medium variability.
An analysis was conducted to determine if the subjects! estimated TOA window differed as a
function of whether the subject thought the speed profile was new or old. For each profile,
the mean TOA window was calculated for those that answered “old/trained” and for those
that answered “new/untrained”. Due to unequal numbers of samples across conditions,
statistical analyses were not conducted, but mean TOA windows are presented in Figure 5-
15 below. Averaged across all speed profiles, the mean estimated TOA window for trials in
which the subject thought the speed profile was "new! was 6.9 (170 trials, SD = 3.9), as
compared to an average of 5.44 (261 trials, SD = 3.7) for trials in which the subject thought
that the speed profile was old. This finding reflects presumably greater uncertainty
associated with an unknown profile. As can be seen in Figure 5-15, the TOA window
estimates for each profile were very similar for those that the subjects thought had a new
speed profile and those which they thought were old, with the exception of two new speed
profile conditions, 45/6 and 45/9. In both cases when subjects correctly identified these as
new profiles, the estimated TOA uncertainty was much larger than when they incorrectly
identified them as old profiles.
84
Figure 5-15. TOA window estimates for each speed profile plotted by subjects! identification of old/trained or new/untrained profile (shown with +/- 1 Std. Error). Parentheses at the bottom indicate whether the speed profile was actually trained or untrained.
Lastly, subjects! estimated TOA windows from Phase 3c were compared to those from
Phase 3b. The only difference between the two conditions was that the speed profiles were
labelled (by colour code) in 3b, but was not labelled in 3c. Figure 5-16 compares the TOA
window estimation error scores for Phase 3b and 3c, for each of the four trained speed
profiles, revealing a phase x profile interaction (F(3,45)=4.92, p=.005). The difference in
TOA window estimation error scores between Phase 3b (labelled) and Phase 3c (not
labelled) was significant only in the slow/high profile condition (F(1,15)=7.24, p=.017), with
estimation error higher when the profile was not labelled (Phase 3c). This result suggests
that there may be an operational benefit to providing display technology or otherwise
identifying the speed profile to operators, since TOA uncertainty estimation error was larger
in the absence of the profile identification. There were only small differences (p<.05)
between the two studies for the other three profiles, suggesting that, for these profiles,
subjects! TOA window estimates were not altered if the speed profile was labelled or
unlabelled.
85
Figure 5-16. TOA window error scores when the profile was colour-coded (phase 3b) and not colour-coded (phase 3c) (shown plotted with +/- 1 Std. Error)
Understanding the process of extrapolating to untrained speed profiles. Further analyses
were conducted to explore how subjects extrapolated from trained profiles to assess TOA
uncertainty for a new speed profile. In particular, given that subjects were trained on low
and high variability conditions, this analysis set out to determine how subjects estimated
TOA windows for the new, untrained, medium variability conditions. Nisbett, Krantz, Jepson
& Fong!s (1982) research on inductive reasoning suggests two mechanisms that could
explain how subjects went about estimating TOA uncertainty for untrained conditions,
yielding two competing hypotheses for consideration. First, subjects could have estimated
the TOA window for low and high variability based on their model of uncertainty developed
in Phase 3a, and averaged the two numbers to determine the TOA window size for the
medium variability condition. This is the simplest form of a mathematical model, or statistical
heuristic (Nisbett, Krantz, Jepson & Fong, 1982), that could reasonably be expected to be
employed by the subjects during the experiment. Alternatively, subjects could have selected
the trained condition that they felt was closest to the new condition, and simply assume the
same TOA window. This is commonly referred to in the literature on judgment under
uncertainty as the representativeness heuristic (Tversky & Kahneman, 1982).
Speed Profile
TO
A W
ind
ow
Err
or
Sco
re
86
To test these two hypotheses, a multiple regression analysis was conducted to determine
which hypothesis better predicted the outcome measure of estimated TOA window for the
new, untrained profiles. The first hypothesis was modelled as a predictor variable that was
computed as the average of each subject!s estimated TOA window for the low and high
variability conditions. The second hypothesis was modelled as a predictor variable that
represented the subject!s estimated TOA window for the speed profile that he/she stated
most closely matched the stimuli. With both predictor variables in the equation, the model
accounted for 80% of the variance in estimated TOA window times.
Referring to Table 5-7, examination of each predictor variable revealed that only the
predictor variable that represented the subject!s estimate for the closest profile was
significant, with a standardized beta score of .878. This suggests that subjects
overwhelmingly employed the representativeness heuristic – that is, they applied the TOA
window of the trained profile that was most similar rather than the average of the low and
high variability. However, the reader must be aware of one potential limitation of this finding.
Because subjects were first asked to identify the closest trained profile and then determine
the TOA window, it is possible that this experimental method cued the subjects to use the
representativeness heuristic.
Table 5-7. Multiple regression analysis to determine which model best explains how subjects estimated TOA uncertainty for untrained profiles.
Model B Beta t sig
Closest Profile .833 .878 11.012 .000
Average of Low and High 0.29 .22 .278 .782
R2 = .804
A subsequent multiple regression analysis was conducted that included the subjects!
similarity rating scores, in addition to the two heuristics as predictor variables. This was
conducted to determine if the degree of rated similarity to a trained profile accounted for any
more variance in estimated TOA window size. Results revealed that it did not – producing a
R2 of .805, as compared to the previously obtained .804. The similarity score predictor
variable was not significant (t=.681, p>.05) with a standardized beta weight of -0.032.
87
5.5 CONCLUSION AND OPERATIONAL IMPLICATIONS
This experiment addressed two methodological limitations identified in experiment 2. The
inclusion of Phase 3a, an extensive training session that exposed subjects to over 100
vehicles for each of four profiles, served to increase the sample size from which subjects
could understand and characterize the TOA distribution, and also changed the task from one
of (potentially) simply observing and describing a single sample of displayed speed
readouts, to one of developing an internal model for application in subsequent phases of the
experiment. Additionally, changes were made to the experimental protocol to improve
subjects! ability to understand the task.
Not only did this experiment replicate the results of experiment 2 (reported in Chapter 4), it
also revealed further important insights into the TOA uncertainty estimation problem. The
results of Phase 3a revealed that using a training-with-feedback paradigm, subjects can
develop accurate models of speed profiles based on relatively short exposure to only 100
vehicles. Phase 3b then asked subjects to apply their models of each speed profile to
estimate TOA uncertainty, for both the trained distance and two new, untrained distances
(one shorter and one longer). For the trained distance, subjects estimated the TOA
Uncertainty for the slow/low profile very accurately, but underestimated TOA Uncertainty for
slow/high while overestimating for the fast/low and fast/high profiles. The untrained distance
conditions revealed systematic biases – overestimating TOA uncertainty for short distances
and underestimating TOA uncertainty for long distances. The TOA uncertainty estimation
error was greatest for the slow/high profile, and this was attributed to subjects! assumption
that the TOA distribution was symmetrical when in fact it is highly positively skewed.
Phase 3c set out to determine how well subjects could identify an unlabeled speed profile by
observing speed fluctuations and then apply the appropriate TOA uncertainty estimate.
Results revealed that subjects could indeed identify the correct profile about 73% of the
time, and consistently applied TOA uncertainty estimates that were similar to estimates of
the labelled vehicles in Phase 3b; however TOA estimation error was higher when the
vehicle was unlabelled in the slow/high profile. Further, Phase 3c examined how subjects
88
estimated TOA uncertainty for new, untrained speed profiles. A series of analyses
suggested that subjects applied a representative heuristic by selecting the trained speed
profile that was most similar to the observed vehicle!s profile, and applying the TOA
uncertainty estimate of that trained profile.
Implications of these results for the proposed model of TOA uncertainty are discussed next
in Chapter 6 and their implications for the operational environment of future 4-D air traffic
control operations are then discussed in Chapter 7.
89
CHAPTER 6: MODEL REFINEMENT
6.1 TIME OF ARRIVAL UNCERTAINTY MODEL RESTATED AND REFINED
The preliminary model of TOA uncertainty, first presented in chapter 2, and reproduced
below in Figure 6-1, was generated based on input from subject matter experts (ATC), after
the first experiment.
Figure 6-1. Original Model of TOA Uncertainty
After exploring the model systematically in two tightly controlled human-in-the-loop
experiments, a proposed modification to the model is presented in Figure 6-2. Results of
experiment 3c revealed that subjects did not change the way they estimated TOA
uncertainty as a function of whether they had seen the profile before or not, as was
previously hypothesised. Rather in all cases, after observing the vehicles, subjects appeared
to determine which of the profiles in their personal library that it most closely matched, and
applied that estimate of TOA uncertainty directly.
Figure 6-2. Refined Model of TOA Uncertainty
90
6.2 SOURCES OF TIME OF ARRIVAL UNCERTAINTY ESTIMATION ERROR
Based on the results of experiments 2 and 3, it is now possible to explain this model further,
by characterizing the sources of TOA uncertainty error and determining how each of the
three processes represented in the refined model of Figure 6-2 (i.e., Build library of speed
profiles; Identify closest speed profile, and Apply estimate of TOA uncertainty) contribute to
TOA uncertainty estimation error observed in study 3c. A multiple regression analysis was
conducted to determine the relative contribution of each of the model processes in
explaining the observed TOA estimation error. In the multiple regression analysis, the
subjects! ability to build a library of speed profiles was modelled using two variables from
study 3a: 1) speed mean estimation error and 2) speed window estimation error. Subjects!
ability to identify the closest speed profile was modelled using two variables from study 3c:
1) error associated with identifying whether the profile was a trained one, or a new one, and
2) error associated with identifying the speed profile for trained profiles only. Finally,
subjects! ability to estimate TOA uncertainty was modelled as TOA uncertainty estimation
errors from study 3b.
The multiple regression revealed that, collectively, the five predictor variables accounted for
75% of the variance. However, as can be seen in Table 6-1, the variable that contributed
most significantly to this was the subjects! TOA uncertainty estimation error score (t=4.65,
p=.001). This suggests that, based on the experimental studies, TOA uncertainty estimation
error can be attributed primarily to the actual TOA estimation process itself, and not to the
process associated with developing the library of speed profiles, or comparing observed
vehicles to the library and identifying the closest profile.
It is prudent to raise one caveat associated with this conclusion. That is, given the
experimental method employed, it was necessary to provide feedback to subjects in study
3a, during which they built their library of speed profiles. While it is believed that this
feedback would be obtained implicitly over time by operators in the actual environment, it is
unknown how long it would take to achieve the same level of performance without the
explicit feedback. Actual operations may yield more error in the first two processes (Building
a library of speed profiles and Identifying the closest profile from the library) than was
observed in these studies.
91
Table 6-1. Multiple Regression to determine source of TOA Uncertainty Error
Variable B Beta t Sig
Speed Mean Estimation Error 0.082 0.032 0.187 0.855
Speed Window Estimation Error 0.203 0.192 1.127 0.286
Old/New Identification 0.034 0.037 0.218 0.832
Speed Profile Identification -0.192 -0.163 -0.914 0.38
TOA Uncertainty Estimation Error for labelled profiles
0.845 0.793 4.65 0.001
R2 = .75
6.3 CAUSES OF TIME OF ARRIVAL UNCERTAINTY ESTIMATION ERROR
Having identified the process of TOA uncertainty estimation as the potentially largest
contributor to error, the next step is to understand the causes of these errors. Further
examination of the results from experiments 2, 3b, and 3c reveal two systematic biases that
contribute to TOA uncertainty estimation errors.
6.3.1 Assumption of Symmetry
First, data in both experiments 2 and 3 revealed that subjects typically failed to recognize
the asymmetrical shape of the TOA distribution and thus underestimated the amount of
actual TOA uncertainty. (Recall, that approximately half of the subjects had completed at
least one university-level course in statistics). As discussed previously, if speed is normally
distributed, then by definition TOA is positively skewed, since TOA is the reciprocal of
speed. Furthermore, this amount of positive skew is more pronounced for vehicles travelling
at slower speeds and with higher variability. However, subjects appeared to treat all
distributions as symmetrical, or in any case to underestimate the amount of positive skew.
This can be seen clearly in the symmetry metric calculated in both experiment 2 (shown
previously in chapter 4) and experiment 3b, shown previously in chapter 5 and reproduced
here in Figure 6-3.
A second piece of evidence that points to the assumption of normality explanation as a
cause of TOA uncertainty estimation error is the speed x variability interaction observed in
92
experiment 3b, shown previously in chapter 5 and reproduced here in Figure 6-4. As can be
seen, subjects estimated the TOA window to be larger for slower vehicles than faster
vehicles, but failed to recognize the sharp increase in TOA window size associated with the
slow/high variability condition due to the positively skewed TOA distribution.
Figure 6-3. TOA window symmetry scores from experiment 3b show that subjects
underestimated the magnitude of positive skew for slow speeds.
93
Figure 6-4. Estimated and actual TOA window size from experiment 3b shows that subjects underestimated the window size for the slow/high condition, which is highly skewed.
(Error bars represent ±95% CI)
Lastly, subjective reports elicited during the debrief interview also support the assumption of
normality hypothesis. Most subjects expressed a strong bias towards symmetrical TOA
windows, as evidenced by the following comments:
• “My TOA windows were always symmetrical, because speed was always
symmetrical.”
• “TOA windows were symmetrical. There is no point in making asymmetrical
windows. Asymmetrical windows would have been strange.”
• “TOA windows were symmetrical, I have an obsession with symmetry.”
• “I thought all TOA windows should be symmetrical.”
• “I made nice even windows around the TOA.”
• “Symmetrical windows just seemed natural.”
In some cases, subjects expressed confusion or uncertainty as to whether the windows
should be symmetrical, positively skewed, or negatively skewed. The following quotes
demonstrated that these subjects did not clearly understand that TOA distributions are
positively skewed:
94
• “As variability increased, the TOA windows tended to be more asymmetrical, but not
always in the same direction. The red profile [slow/low] had more early vehicles, and
the blue profile [fast/high] had more later vehicles.”
• “My TOA windows tended to be asymmetrical – sometimes bigger on the left [more
earlier than later] and sometimes bigger on the right [more later than earlier]. I don!t
know why.”
However two subjects did recognize that TOA windows were asymmetrical, as evidenced by
the following quotes:
• “ The TOA windows were symmetrical, except for the yellow profile [slow / high
variability] because they lagged a bit.”
• “TOA windows were asymmetrical. In actuality, there is no way that arrivals would
be symmetrical – there is more of a chance of being slow and taking longer.”
6.3.2 Aversion to Extremes
A second potential source of TOA Uncertainty estimation error was an apparent aversion to
extremes. In both experiment 2 and 3, subjects overestimated conditions of small
uncertainty and underestimated conditions of large uncertainty. This systematic bias can be
seen in the speed x variability interaction observed in both experiment 2 and 3b shown
above in Figure 6-4. Subjects overestimated TOA uncertainty for the fast speed profiles that
possessed low actual TOA uncertainty (fast / low variability and fast / high variability) and
underestimated TOA uncertainty for the slow profiles which possessed higher actual TOA
uncertainty. Finally this can also be seen in the adjusted distance extrapolation results of
experiment 3b, reproduced here in Figure 6-5. Subjects systematically overestimated the
low actual TOA uncertainty of short distances and underestimated the higher actual TOA
uncertainty of longer distances. This was seen for each of the four speed profiles. This
systematic aversion to extremes is compatible with previous research on the estimation of
probabilistic events conducted by Tversky and Simonson (2000), which suggests that most
people tend to avoid extremes and choose an intermediate option if available.
95
Figure 6-5. TOA window estimates as a function of speed profiles for trained and untrained distances, showing subjects! systematic bias for overestimating low uncertainty and underestimating high uncertainty.
96
CHAPTER 7: SUMMARY, LIMITATIONS AND IMPLICATIONS
The ability to estimate TOA uncertainty is a critical cognitive process in supervisory control
systems such as air traffic management. It is expected to become a critical issue for the next
generation aviation system, which will rely on strict conformance and monitoring of 4-D
trajectories both in the air and on the ground. Chapter 1 discussed the introduction of new
decision support system technologies to support 4-D operations. There the need for
controllers to develop effective estimates of TOA uncertainty rather than relying on
automation for this purpose was identified.
In chapter 2, experiment 1 solicited input from ATC subject matter experts to shed light on
how TOA is estimated in the real world, together with the uncertainty associated with those
estimates. Several factors believed to impact TOA uncertainty estimates were identified.
This experiment led to the development of a preliminary model of TOA uncertainty
estimation.
In chapter 3, literature from domains such as decision making under uncertainty and
probabilistic reasoning was applied to develop a generic experimental method and metrics
for evaluating TOA uncertainty, extracted from relevant aspects of the ATC TOA estimation
task. This experimental method was then applied in experiment 2 in chapter 4, which
identified that subjects tended to over-estimate TOA uncertainty for fast speeds, and
underestimate it for slow speeds. The latter was attributed to underestimating the amount of
positive skew associated with the TOA distribution.
In chapter 5, the results of experiment 3 addressed two limitations of the experimental
paradigm and replicated the results of experiment 2. Experiment 3 also extended the
findings to better understand how subjects estimate TOA uncertainty for untrained distances
and profiles. From experiment 3, it was concluded that subjects appeared to apply the
representativeness heuristic when estimating TOA uncertainty for new speed profiles that
they have not previously encountered.
97
7.1 MODEL ASSUMPTIONS AND LIMITATIONS
The refined model of TOA uncertainty estimation presented in chapter 6, along with the
sources and causes of estimation error, are based on certain assumptions as follows.
7.1.1 Speed is Normally Distributed
First, the model and results are predicated on the assumption that speeds are normally
distributed, which was the case in the two experimental studies. Although this is a
simplification of the operational environment, data from a recent airport human-in-the-loop
taxiing simulation (Williams, Hooey & Foyle, 2006) nevertheless support that this is the case
for nominal taxi conditions. Figure 7-1 presents a histogram of a short sample of taxi speed
data taken from that study, in which the pilot was taxiing in a medium-fidelity aircraft
simulator while trying to maintain a commanded average taxi speed, unencumbered by
traffic delays. As can be seen, those taxi speeds do approximate a normal distribution in this
simple taxi condition. Of course, this would not necessarily hold in more complex scenarios
with traffic and adverse weather. In addition, it is unknown whether the subjects understood,
or assumed, that the distribution of speeds was normally distributed. It is indeed possible,
for example, that subjects envisioned a uniform distribution in which case they would not
necessarily expect the resulting TOA distribution to be positively skewed. This could have
accounted for the subjects! underestimates of positive skew.
Figure 7-1 Taxi speed data from a medium fidelity pilot-in-the-loop study, showing that
nominal taxi speed maintenance data are approximately normally distributed.
98
7.1.2 Availability of Accurate Speed Readouts
Second, the model assumes that operators have access to speed data, such as in the form
of a digital readout, and are not required to perceive changes in speed visually. In many
ways this simplifies the task and the model, but in a manner that is consistent with future
aviation operations. It is a safe assumption that both pilots and controllers will be equipped
with accurate speed-readouts to assist them in advanced 4-D trajectory environments.
However, there are potential implications for environments where this assumption may not
hold true. Estimates of TOA uncertainty may be systematically influenced by subjects!
misperception of vehicle speed that may occur as a function of the stimulus field, object size,
and the background texture.
As summarized by Ryan and Zanker (2001) and Goldstein (1989), perception of speed is
affected by the size of both the moving object being observed and the framework through
which it moves. The perceived speed of regularly spaced objects moving within apertures
decreases with larger aperture size (Brown, 1931). In particular, Brown asked observers
sitting in a dimly lit room to adjust the speed of a large dot moving across a large rectangle
to make it appear equal to the speed of a small dot moving across a small rectangle. Brown
found that, if the large rectangle was ten times larger than the small one, the large dot had to
move seven times faster than the small one for them to appear to move at the same speed.
This effect, known as speed transposition, shows that two images moving across the visual
field at different speeds can be perceived to be moving at the same speed. This could be
due to a size constancy effect by which, in the absence of veridical depth information, a
smaller stimulus is interpreted as being further away from the observer. This could have
important implications in some domains, particularly if the user is permitted to resize
windows on their display, thus making objects appear to be moving faster or slower. This
effect would be particularly problematic given the results found here, which show that
subjects tend to overestimate TOA uncertainty for faster objects and underestimate TOA
uncertainty for slower objects.
Another factor that may affect the perception of speed is the background on which the object
is displayed. This may range from objects in the real world, to icons on a display with
backgrounds that consist of simple, solid colour backgrounds, to complex grid backgrounds.
99
The threshold for perceiving movement in a homogeneous field is a speed of about 1/6 to
1/3 of a degree of visual angle per second (Goldstein, 1989). However, if vertical lines are
present in the background, the object!s movement can be perceived even at speeds as low
as one-sixtieth of a degree of visual angle per second. Thus, perception of motion is
determined by the background, which affects not only the threshold for movement but also
the perception of speed. Indeed, the grid marks placed on the vehicle!s route in experiment
3 (see chapter 5) could have caused subjects to perceive the speed to be greater than it
really was. Related to this, it is also unknown if the perception of velocity was uniform
across the display (as opposed to the possibility that higher speeds might have been
perceived at the edges of the display), and precisely how this may have affected the results.
However, it is important to note that the factors outlined above are unlikely to have
confounded the results in the current experiments, for two reasons: 1) Subjects in these
experiments relied primarily on the digital speed readouts to estimate speed and thus did so
with negligible perceptual error; and 2) Speed estimation performance in experiment 3 was
very similar to that in experiment 2, in which there were no vertical hatch marks along the
route. Nevertheless, these factors could substantially influence speed perception in
operational settings in which operators are provided displays that show vehicles moving
against a background that changes from visually simple to complex as a function of the
environment such as a road network or airport layout.
7.1.3 Time of Arrival Uncertainty Estimation Adheres to the Proposed Two-step
Process Model
The model and experiments were also predicated on the assumption that subjects first learn
speed profiles and for each speed profile they estimate an expected TOA uncertainty
window. Indeed, this assumption was generated based on results of the first study in which
air traffic controllers described how they estimated TOA and TOA uncertainty. Controllers
reported that they had mentally assigned certain airlines with a speed profile (e.g.,
“Southwest Airlines taxied fast and consistently”) and estimated TOA based on that profile.
It is for this reason that in experiment 3 subjects were trained (with feedback) on speed and
then tested on their ability to estimate TOA. If, instead, subjects had been trained to
estimate TOA by means of a training protocol that provided feedback on TOA instead of
speed, the results most likely would have yielded improved TOA uncertainty estimates.
100
Nonetheless, it is believed that operators, such as air traffic controllers, do indeed develop
models of speed profiles from which they estimate TOA, rather than developing models of
TOA directly.
Nonetheless, this is an important assumption to consider and raises implications for real-
world operational training and display needs. In the operational setting, there are four
important differences from the studies presented in this research: 1) controllers don!t always
get immediate feedback in terms of their speed and TOA estimates; 2) the on-the-job
training would not provide feedback that is as precise and unambiguous as in experiment 3;
3) controllers won!t likely experience 100 aircraft in a row of the same profile, and thus may
experience degradations of their estimates due to interruptions and integration of multiple
profiles; and 4) controllers would need to generate many more speed profiles to capture the
range of operational conditions and the differences among these profiles may be more
subtle than those tested in experiment 3. Exactly how these differences help or hinder
controllers! ability to build a library of speed profiles and apply them to estimate TOA
uncertainty is a topic for further research using operational settings and experienced
controllers.
7.2 OPERATIONAL IMPLICATIONS FOR AVIATION
A number of important implications for the operational environment have been revealed.
These implications are reported below, with the caveat that, without further operational or
field research, the degree to which these laboratory-based studies generalize to expert air
traffic controllers in the operational environment is unknown.
7.2.1. Implications for Human Error
First, it was shown that subjects underestimate TOA uncertainty in conditions where TOA
objective uncertainty is high, such as aircraft travelling long distances and aircraft that travel
at slow average speeds that are highly variable. This may have implications for actual
aviation operations, in that aircraft in these operating conditions may be sequenced too
closely because controllers will think the aircraft need less separation than they actually do.
In the example of runway sequencing, it is likely that controllers will produce smaller-than-
101
necessary runway-crossing windows (a gap between two arriving aircraft in which a third
aircraft can cross the active runway). That is, if a controller creates a runway-crossing
window that is based on an underestimated TOA uncertainty window, the resulting runway-
crossing window will be too small. It is likely that the second arriving aircraft will be required
to conduct a go-around manoeuvre because the crossing aircraft will not have cleared the
runway in time. This could be a safety hazard, leading to potential loss-of-separation, or
create system inefficiencies if aircraft need to be stopped and/or re-routed.
Second, the subjects underestimated the amount of positive skew of TOA distributions for
slow and highly variable aircraft, thus a higher number of aircraft arrived later than expected.
This implies that if actual controllers also exhibit this tendency, they will underestimate the
likelihood that an aircraft will arrive late at a destination (e.g., waypoint, runway, or gate).
This can have both safety and efficiency implications in the case of paired departure
operations (Lunsford, 2009), in which two aircraft must depart parallel runways within 30
seconds of each other to ensure the following aircraft is not subject to excessive wake
turbulence from the lead aircraft. If controllers underestimate the likelihood that the following
aircraft may arrive late at the departure runway, the following aircraft may either experience
a wake vortex incident or experience substantial delays, as it would be required to stop and
wait for the next available departure slot, resulting in system inefficiencies.
Third, subjects overestimated TOA uncertainty when the distance was short, and when
actual TOA uncertainty was low, as with aircraft that travel at a consistently fast speed. If
actual controllers were also to exhibit this tendency, the result would be larger-than-
necessary runway crossing windows. That is, for example, a controller might leave a gap of
two minutes between arriving aircraft to ensure that a third aircraft can cross the active
runway, when in fact a gap of only one minute is required. With accurate knowledge of TOA
uncertainty, the controller could have doubled runway-crossing throughput by crossing two
aircraft in the gap, instead of one. In these cases, overestimates of TOA uncertainty reduce
airport efficiency. Also, in this case, a controller may unnecessarily require an arriving
aircraft to conduct a go-around manoeuvre if he/she believes the runway will not be clear,
which creates additional risks for the arriving aircraft.
102
Finally, subjects tended to generalize unknown profiles to the closest known profile. If
controllers do this in the operational environment, and there is evidence from experiment 1
that they do, this could lead to over-generalizations, resulting in TOA windows that are either
too small or too large as described above, again resulting in either unsafe or inefficient
operations.
7.2.2 Implications for Human-Centred Automation to Support Air Traffic Control
Given the prevalence of the TOA uncertainty estimation errors noted above, it is next
reasonable to consider how the performance of air traffic controllers, particularly in next-
generation time-based environments, could be supported by automation. The results of this
research are interpreted using Parasuraman, Sheridan, and Wickens! (2000) "stages and
levels! of automation framework. The framework proposes that automation can support four
broad classes of functions: 1) Information Acquisition; 2) Information Analysis; 3) Decision
and Action Selection; and 4) Action Implementation. Within each of these functions,
automation can be applied across a continuum of levels from low to high (i.e., from fully
manual to fully automatic). Suggestions for how automation can be used to support air
traffic controllers in each stage are discussed next.
Information Acquisition refers to the process by which raw data are selectively attended or
filtered (Wickens, 2000). In the context of TOA uncertainty estimation, this includes the
controllers! ability to accurately perceive the aircraft!s current state (location, speed, distance
to travel), identify the factors that contribute to a particular aircraft!s TOA uncertainty, and to
accurately characterize the aircraft!s speed profile. The present research demonstrated
human limitations associated with the task of accurately characterizing an aircraft!s speed
profile. Specifically, it was noted that subjects were able to accurately identify a speed
profile after a short period of observation only 65 to 88% of the time for trained profiles.
Automation to support this information acquisition task may take quite different forms
ranging from simple displays of raw data (particularly speed; see section 7.1.2 regarding the
assumption of this minimum level of automation) to target cueing (Yeh, Wickens, & Seagull,
1999), intelligent information management (Hammer, 1999), and highlighting (Wickens,
Kroft, & Yeh, 2000), each of which explicitly directs the controller!s attention to certain
103
information sources at the expense of others. The automation may guide attention to
sources of environmental information that the controller may use to determine the aircraft!s
speed profile (such as information about traffic, weather, aircraft type, and pilot familiarity
with the airport) or may explicitly provide an indication of an aircraft!s speed profile, possibly
through color-coding on the ATC display.
In the second stage (Information Acquisition), data must be integrated and interpreted so as
to draw some intelligent inference regarding the state of the world (Wickens, 2000). In ATC
operations, this refers to the controllers! estimation of TOA uncertainty based on their library
of speed and TOA profiles. As was seen in the results of experiment 2 and 3, this was the
task that was the most prone to bias, and thus presumably may benefit the most from
automation. To support this task, automation could compute and display TOA uncertainty
and prioritize or rank order the aircraft in terms of TOA uncertainty. A controller could use
the display to choose the aircraft that is most likely to arrive at the departure runway on time,
or to identify aircraft that are unlikely to make their takeoff time.
The third phase (Decision and Action Selection), in the context of ATC, refers to the
controllers! task of deciding whether or not the estimated TOA uncertainty is acceptable, or if
action is required to reroute aircraft. Although not addressed in the current research, there
is evidence that controllers may typically apply a conservative bias in this decision-making
phase (see Boudes & Cellier, 2000) and thus may be more likely to determine that the TOA
uncertainty is unacceptable and reroute aircraft. Although this conservative bias is actually
quite adaptive in today!s ATC environment (as described previously in section 1.1.1), it may
be detrimental in NextGen 4-D environments, which will require aircraft to arrive "just-in-time!
(that is, neither early nor late). Automation in the form of visual or auditory alerts that
indicate when objective TOA uncertainty exceeds a pre-determined threshold could be
applied to support this stage.
Finally, the Action Execution stage in this ATC context refers to the task of issuing a revised
clearance to one or more aircraft as a result of excessively high TOA uncertainty. A lower
level of automation to support this stage could take the form of automation that generates
time-based clearances that the controller would approve or modify and then communicate to
104
the aircraft. This approach would take advantage of the automation!s ability to quickly
compute accurate TOA!s and distributions, but at the same time take advantage of the
human!s ability to improvise and use flexible procedures (see Fitts, 1951). A higher level of
automation could operate completely autonomously and issue time-based clearances
directly to the aircraft and modify them when pilots fail to meet their time requirements. The
controller could over-ride the automation to provide certain aircraft with more time and a
larger uncertainty buffer where mandated. However, it must be acknowledged that a
weakness of this approach is the operator "out-of-the-loop! problem (Endsley & Kris, 1995).
As discussed previously in Section 1.1.3, when placed in a monitoring role, it is often difficult
for the operator to maintain sufficient situation awareness to make the necessary
interventions.
While automation to support the Action-Execution stage is currently being researched (for
example, Balakrishnan & Jung, 2007; Rathinam, Montoya & Jug, 2008), many of these
prototype systems typically address TOA uncertainty in an overly simplistic manner by either
applying a constant uncertainty value for each aircraft or assuming perfect on-time
performance from each aircraft. The current research showed that the TOA uncertainty
values incorporated into the automation should consider (at least) the mean speed,
variability, and distance to travel. Experiment 1 also uncovered several other factors that
human controllers consider in estimating TOA uncertainty, such as airline, weather
conditions, traffic levels, and even whether or not the pilot spoke with an accent (suggesting
to the controllers that the pilot might not be familiar with the airport surface). While some of
these could presumably be incorporated into the automation algorithms (with varying
degrees of success) others, such as the pilot!s accent, are not easily considered by
automation. Failure to account for these important variables may result in inaccurate
uncertainty buffers around each aircraft. Uncertainty buffers that are too large will limit
system efficiency and uncertainty buffers that are too small may yield excessively high error
rates.
In summary, automation that supports controllers in the Information Acquisition task may be
the most fruitful for several reasons. First, as noted above, subjects exhibited systematic
biases in their abilities to estimate TOA uncertainty. Second, applying automation at the
earlier stages, rather than at the decision and action stages, may indeed be the best option
105
if the airport surface environment proves to be too dynamic, thus rendering the automation
"brittle" (Cohen, 1993; as discussed previously in section 1.13). This approach would
minimize the operator out-of-the-loop concern because the controller would be actively
controlling the airport traffic, albeit in a more informed manner. Third, a further potential
advantage of automation at the earlier stages that were shown to exhibit systematic bias, is
that it is likely that controllers! TOA uncertainty estimation errors would gradually correct
over time with use of the automation and their estimates would become more accurate.
Indeed, such an improvement in estimation error was observed in Experiment 3 for the task
of velocity and velocity window estimation errors. It is plausible that displaying the TOA
uncertainty information to the controller would serve as adequate feedback to support this
learning process in the actual operational environment. If this is the case, it is possible that
the controllers would rely less on the automation over time, and use it as an aid only when
presented with an aircraft with an unfamiliar profile.
The results of the current research suggest that automation, particularly when applied to
Stages 1 and 2, could help compensate for the systematic biases observed in the tasks
associated with identifying an aircraft!s speed profile and applying the TOA estimate. It is
still unclear, however, exactly how a controller would, or should, use such TOA uncertainty
information, and this remains a topic for future research.
7.3 FUTURE DIRECTIONS
This research culminated in the development of a model that explains how subjects
estimated TOA uncertainty, and the sources and causes of errors in the TOA uncertainty
estimation process. Armed with this model of the systematic biases inherent in human
operators! TOA uncertainty estimation, three future research directions have been identified.
First, the results of the current research were based on a simplification of a real-world task.
Although it is believed that the important elements of the task were represented in the
experimental tasks reported in chapters 4 and 5, future research is required to increase the
realism and operational robustness of the experimental task, to ensure that the findings
generalize to real world ATC operations. This could include modifications to the current
experimental platform, such as including variable start times and route lengths,
106
representations of traffic sequencing tasks and other environmental cues that affect
variability, and implementing variable profiles. Future analyses of actual operational data in
an ATC setting would be of value, as would modelling profiles with realistic, potentially non-
normal distributions.
Second, various forms of ATC automation were presented and discussed in section 7.2.2.
Each of these approaches requires extensive research including human-in-the-loop
simulations to fully define and evaluate. Only with relatively mature automation solutions (or
at least emulations) can the relative merits of each automation level for a particular stage be
evaluated.
Third, given that TOA uncertainty exists in many complex environments and humans
systematically misestimate this uncertainty, there is a need for further research and
development efforts geared toward the design and development of DSSs and other
advanced visual displays to support users in the TOA uncertainty estimation task (such as
Lee & Milgram, 2008). It is expected that elements of the experimental method and metrics
developed within the current research effort will be useful for this purpose.
Fourth, the current research has examined how subjects comprehend TOA uncertainty
relative to actual TOA uncertainty; however, there is still much to be learned regarding what
users will do with this information. Future research is required to better understand how this
comprehension of TOA uncertainty will affect operator situation awareness and the actual
decisions that operators make.
107
CHAPTER 8: REFERENCES
Alpert, M. & Raiffa, H. (1982). A progress report on the training of probability assessors. In
D. Kahneman, P. Slovic and A. Tversky (Eds.). Judgment Under Uncertainty:
Heuristics and Biases. Cambridge: Cambridge University Press.
Andre, A. D. & Cutler, H. A. (1998). Displaying uncertainty in advanced navigation systems.
Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting. (pp.
31-35). Santa Monica, CA: HFES.
Balakrishnan, H., & Jung, Y. (2007). A framework for coordinated surface operations
planning at Dallas-Fort Worth International Airport (Paper # AIAA 2007-6553).
Proceedings of the AIAA Guidance, Navigation, and Control Conference and Exhibit.
Alexandria, VA: AIAA
Banbury, S., Selcon, S., Endsley, M. Gorton, T. & Tatlock, K. (1998). Being certain about
Banbury, S., Selcon, S., Endsley, M. Gorton, T. & Tatlock, K. (1998). Being certain
about uncertainty: How the representation of system reliability affects pilot decision
making. Proceedings of the Human Factors and Ergonomics Society 42nd Annual
Meeting. (pp. 36-39). Santa Monica, CA: HFES
Block, R. A. & Harper, D. R. (1991). Overconfidence in estimation: Testing the anchoring
and adjustment hypothesis. Organizational Behavior and Human Decision Processes,
49, 188-207.
Boudes, N. & Cellier, J-M. (2000). Accuracy estimations made by air traffic controllers. The
International Journal of Aviation Psychology, 10(2), 207-225.
Brown, J. F. (1931). The visual perception of velocity. Psychologische Forschung, 14, 199-
232.
Cohen, M. S. (1993). The bottom line: Naturalistic decision aiding. In G. A. Klein, J.
Orasanu, R. Calderwood, & C. E. Zsambok (Eds.). Decision Making in Action: Models
and Methods. Norwood, NJ: Ablex Publishing Corporation.
108
Endsley, M. R., Bolte, B. & Jones, D. G. (2003). Designing for Situational Awareness: An
approach to user-centered design. New York: Taylor & Francis.
Endsley, M. R., & Kris, E. O. (1995). The out-of-the-loop performance problem and level of
control. Human Factors, 37 (2), 381-394.
Erev, I., Wallsten, T.S., & Budescu, D. V. (1994). Simultaneous over- and underconfidence:
The role of error in judgment processes. Psychological Review, 101, 519-527
Fitts, P. M. (1951). Human Engineering for an Effective Air Navigation and Traffic Control
System. Washington DC: National Research Council.
Gigerenzer, G., Hoffrage, U., & Kleinbolting, H. (1991). Probabilistic mental models: A
Brunswikian theory of confidence. Psychological Review, 98, 506-528.
Goldstein, E. B. (1989). Sensation and Perception. Belmont, California: Wadsworth
Publishing Company.
Hammer, J. M. (1999). Human factors of functionality and intelligent avionics. In D. J.
Garland, J. A. Wise, & V. D. Hopkin (Eds.), Handbook of Human Factors in Aviation. (pp.
549-565). Mahwah, NJ: Lawrence Erlbaum.
Hansman, R. J. & Davison, H. J. (2000). The effect of shared information on pilot / controller
and controller / controller interactions. 3rd USA/Europe ATM R&D Seminar, Napoli, Italy.
Johnson, W.W., Battiste, V. & Holland, S. (1999) A cockpit display designed to enable
limited flight deck separation responsibility. Proceedings of the 1999 World Aviation
Conference, Anaheim, CA.
Juslin, P. (1994). The overconfidence phenomenon as a consequence of informal
experimenter-guided selection of times. Organizational Behavior and Human Decision
Processes, 57, 226-246.
Juslin, P., Wennerholm, P., & Olsson, H. (1999). Format dependence in subjective
probability calibration. Journal of Experimental Psychology: Learning Memory and
109
Cognition, 25(4), 1038-1052.
Juslin, P., Winman, A., & Olsson, H. (2003). Calibration, additivity, and source
independence of probability judgments in general knowledge and sensory discrimination
tasks. Organizational Behavior and Human Decision Processes. 92, 34-51.
Kahneman, D., Slovic, P. & Tversky, A. (1982). Judgment under Uncertainty: Heuristics and
Biases. NY: Cambridge University Press.
Kuchar, J. K. (2001). Managing uncertainty in decision-aiding and alerting systems.
Proceedings of the 6th CNS/ATM Conference. Taipei, Taiwan.
Lee, H. & Milgram. P (2008). An Integrated Air Traffic Control Display Concept for Conveying
Temporal and Probabilistic Conflict Information. Proceedings of the Human Factors and
Ergonomics Society (pp. 84-84). Santa Monica, HFES.
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982). Calibration of subjective probabilities:
The state of the art up to 1980. In D. Kahneman, P. Slovic, & A. Tversky (Eds.),
Judgment under Uncertainty: Heuristics and Biases (pp. 306-334). New York:
Cambridge University Press
Lunsford. C. (2009). Feasibility and availability of pairing departures from closely spaced
parallel runways for wake avoidance. Proceedings of the 2009 IEEE/AIAA Integrated
Communications, Navigation and Surveillance Conference (ICNS) conference.
Arlington, VA, May 2009.
McKenzie, C. R. M., Liersch, M. J., & Yaniv, I. (2008). Overconfidence in interval estimates:
What does expertise buy you? Organizational Behavior and Human Decision
Processes, 107, 179-191.
Nisbett, R. E., Krantz, D. H., Jepson, C. & Fong, G. T. (1982). Improving inductive
inference. In D. Kahneman, P. Slovic, & A. Tversky (Eds.). Judgment under
uncertainty: Heuristics and biases. Cambridge: Cambridge University Press
O!Connor, M., & Lawrence, M. (1989). An examination of the accuracy of judgemental
confidence intervals in time series forecasting. Journal of Forecasting, 8(2), 141-155.
110
Paese, P. W., & Sniezek, J. A. (1991). Influences on the appropriateness of confidence in
judgment: Practice, effort, information, and decision making. Organizational Behavior
and Decision Making, 48, 100-130.
Peterson, C. R. & Beach, L. R. (1967). Man as an intuitive statistician. Psychological
Bulletin, 68(1), 29-4.
Pitz, G. F. (1980). The very guide of life: The use of probabilistic information for making
decisions. In T. S. Wallsten (Ed.) Cognitive Processes in Choice and Decision.
Hillsdale, N.J: Erlbaum and Associates.
Rathinam, S., Montoya, J., & Jung, Y. (2008). An optimization model for reducing aircraft
taxi times at the Dallas Fort Worth International Airport. Proceedings of the 26th
International Congress of the Aeronautical Sciences (ICAS). Sept 14-19, 2008,
Anchorage, Alaska.
Ryan, J. & Zanker, J. M. (2001). What determines the perceived speed of dots moving
within apertures? Exp Brain Res. 141, 79-87.
Schaefer, D., Gizdavu, A., & Nicholls, D. (2004). The display of uncertainty information on
the controller working position. Proceedings of the Digital Avionics Systems Conference.
St. Louis, Mo: IEEE.
Schiff, W. & Oldak, R. (1990). Accuracy of judging time to arrival: effects of modality,
trajectory, and gender. J Exp Psychol Hum Percept Perform. 16(2). 303-16.
Schunn, C. D., Kirscehnbaum, S. S., and Trafton, J. G. (2005). The ecology of uncertainty:
Sources, indicators, and strategies for informational uncertainty. Technical Report
(Available at www.au.af.mil/au/awc/awcgate/navy/nrl_uncertainty_taxonomy.pdf)
Sheridan, T. B. (2002). Humans and Automation: System Design and Research Issues.
Santa Monica, CA: HFES / John Wiley& Sons.
Soll, J. B. & Klayman, J. (2004). Overconfidence in Interval Estimates. Journal of
Experimental Psychology: Learning, Memory, and Cognition. 30(2), 29-314.
111
Teigan, K. H., & Halberg, A., & Fostervold, K. I. (2007). More than, less than, or minimum,
maximum: How upper and lower bounds determine subjective interval estimates.
Journal of Behavioral Decision Making, 20, 179-207
Tversky, A. & Kahneman, D. (1982). Judgment under uncertainty: Heuristics and biases.
In D. Kahneman, P. Slovic, & A. Tversky (Eds.). Judgment under uncertainty:
Heuristics and biases. Cambridge: Cambridge University Press
Tversky, A. & Simonson, I. (2000). Context-dependent preferences. In D. Kahneman and
A. Tversky (Eds). Choices, Values, and Frames. Cambridge: Cambridge University
Press
van Doorn B., Bakker, B., & Meckiff, C. (2001). Evaluation of advanced conflict modelling in
the highly interactive problem solver. Proceedings of the 3rd USA/ Europe ATM R&D
Seminar, Napoli.
Xu, A., Wickens, C. D., & Rantan, E. (2004). Imperfect conflict alerting systems for the
cockpit display of traffic information. Technical Report AHFD-04-8/NASA-03-2. Savoy,
Illinois: University of Illinois at Urbana-Champaign, Aviation Human Factors Department.
Wickens, C. D. (2000). Imperfect and unreliable automation and its implications for attention
allocations, information access, and situation awareness. Technical Report ARL-00-
10/NASA-00-02. Savoy, Illinois: ARL, University of Illinois at Urbana-Champaign.
Wickens, C. D., & Hollands, J. G. (2000). Engineering Psychology and Human
Performance. London: Prentice Hall.
Wickens, C. D., Kroft, P., & Yeh, M. (2000). Database overlay in electronic map design:
Testing a computational model. Proceedings of the IEA 2000/HFES 2000 Congress.
Santa Monica, CA: HFES.
Wickens, C. D., & Morphew, E. (1997). Predictive features of a cockpit traffic display: A
workload assessment (Technical Report ARL-97-6/NASA-97-3). Savoy, IL: University
of Illinois, Institute of Aviation, Aviation Research Lab.
112
Williams, J. L., Hooey, B. L., & Foyle, D. C. (2006). 4-D taxi clearances: Pilots' usage of
time- and speed-based formats. Proceedings of the AIAA Modelling and Simulation
Technologies Conference. Paper AIAA-2006-6611.
Yeh, M., Wickens, C. D., & Seagull, F. J. (1999). Target cueing in visual search: The effects
of conformality and display location on the allocation of visual attention). Human Factors,
41, 524-542.
113
APPENDICES
APPENDIX A1: EXPERIMENT 1 PROTOCOL
As we look ahead to the future and the potential implementation of time-based taxi clearances, we acknowledge that the current ground control system is very dynamic and possesses many sources of uncertainty. The purpose of this study is to explore factors that contribute to your perceptions of certainty (or confidence) that a time-based taxi plan that you issue to pilots will be carried out as scheduled. In particular, I am interested in how a number of factors influence your perception of certainty. 1) Distance / Speed Calibration
Before we begin, I will start by providing you with some information to calibrate you to the time required to taxi different routes at two different speeds. You can safely assume that typical taxi speeds range from 16 to 24 knots, with an average speed being 20 knots. This shows two routes running east/west across the airport. One is 6,000! and the other is 12,000!. These are intended to give you a rough idea of how long each taxi route would take to complete at average speeds of 16 kts, 20 kts, and 24 kts. You can see that the distance from one inside runway to the other is about 6,000! and a 12,000! route stretches across most of the airport. Here are the same length routes, this time running north/south on the airport. 2) Scenario Descriptions
In this study, you will see several static pictures of taxi plans in progress. Each picture will show you the taxi plan and an aircraft!s progress in carrying out the plan. Your task will be to assess the information and provide a rating of how certain you are that the plan will succeed (that is, that the aircraft will arrive at the runway threshold within the provided window of time – neither too early, nor too late). Each scenario will contain the following features:
Magenta taxi route: In all trials, you see a map of DFW airport with one taxi route shown in magenta.
Aircraft Icon: The aircraft icon indicates the position of the taxiing aircraft along the route of the aircraft. The aircraft will either be positioned at the beginning of the route or part way through. If it is at the beginning, you can assume that the aircraft has spooled up to a nominal taxi speed.
Data tag: Associated with each aircraft is a data tag with a TIME. This refers to the elapsed time since the pilot started taxiing.
Runway Arrival Time Window: All taxi routes will end at a runway. A runway arrival time window will be provided in the yellow box beside the runway. This time window indicates the acceptable window of time in which the aircraft may arrive at the runway. It DOES NOT include time required to cross the runway.
114
Block 1
You have seen the full range of the factors for this experiment. When you are ready,
we!ll continue with more trials.
Remember to consider all of the factors together and to use the entire scale from 1 (if you have LOW certainty that the plan will succeed) to 7 (if you have HIGH certainty that the plan will succeed). For the remaining trials, use the numbers 1 to 7 on the keyboard to enter your answers. Do you have any questions before we begin? Things to Remember…
1) Please rate how certain you are that the aircraft will arrive at the runway threshold within the cleared window of time using the following scale.
Very Low Certainty Very High Certainty 1 2 3 4 5 6 7
2) Try to use the entire range of the seven-point scale. 3) Runway time windows refer to the arrival time at the runway, NOT the time to cross the runway. 4) The aircraft should arrive at the runway within the crossing window and neither late, nor
early. - Arriving early will require the aircraft to stop – this doubles the time required for an aircraft to cross a runway as compared. - Arriving late will mean the aircraft has missed the crossing window – this will inevitably cause a delay.
5) Assume that it is a clear day at Dallas Forth Worth and there are no traffic conflicts. 6) Assume that each aircraft has navigation and time-management technology that will enable them to arrive at the runway threshold within a 30 second time window about 50% of the time. 7) Assume that an average taxi speed is 20 knots and that the aircraft is a B-757. 8) Remember that every pilot and aircraft is different and any number of factors may influence their ability to comply with the precise time window. For example:
- A pilot may be slow to start taxi or spool up. - A pilot may slow down more for turns than another. - A pilot may be unfamiliar with the airport. - A pilot may have to stop because a passenger is standing up. - There may be equipment problems that need investigating during taxi.
115
APPENDIX A2: EXPERIMENT DEBRIEF INTERVIEW QUESTIONS
1. Please rank the importance of the following six factors in determining your level of certainty (or confidence) that a time-based taxi clearance will be carried out successfully. _____ Route Length _____ Route Complexity _____ Aircraft Position Along Route _____ Duration of Runway Arrival Window 2. What other factors contribute to your certainty that a pilot will arrive at the cleared runway within a specified time window? 3. Please rank these factors relative to the factors in question 1. 4) What strategy did you use for assigning certainty ratings? - At what rating would you cancel a clearance or re-route due to lack of confidence? 5) How did you use the runway threshold window.
- i.e., consider if they would make the middle of the window, the beginning, or the end? 6). What percentage, if any, of your clearances today have some sort of time element in them (i.e. expedite etc.). 7) How do you manage uncertainty in today!s surface environment 8. If you are uncertain that an aircraft will comply with its clearance, how does this impact your task of controlling multiple aircraft at the same time. 9. If you have two aircraft, one that you are certain will comply with its clearance, and one that you are less certain, do you treat monitor them differently? If so how? 10. How will time-based clearances change the way you monitor airport traffic? 11. Do you have any suggestions for managing uncertainty in future time-based environments? - changes to communication, information available, technology?
116
APPENDIX B1: EXPERIMENT 2 PROTOCOL
Welcome and thank you for agreeing to participate in this study. The purpose of this study, which will also form a portion of a PhD research project at the University of Toronto, is to understand how people estimate Time-of-Arrival Uncertainty. Please take a few minutes and read this and the attached informed consent form. It describes the study you are about to participate in, and explains your rights as a subject. If you have any questions at any time, please ask the experimenter. Overview
In this study, I am interested in how you estimate time-of-arrival uncertainty. To start with, I would like you to consider the following scenarios: Scenario A: You travel 10 miles on the freeway to get to work every day for 100 days. Since you work the night shift you don!t have much traffic to worry about, and you always try to stick to the speed limit. Sometimes you go slightly slower or faster when you encounter other traffic along the way, but usually only by 1 or 2 miles per hour. On average, across the 100 days, it takes you 10 minutes to get to work, but of course it is not always exactly 10 minutes because your speed varies slightly, depending on traffic. Scenario B: You travel 10 miles on city streets to get to work every day for 100 days. Since you are always running late, you like to travel as fast as you can, but of course always stopping for red lights. On average, across the 100 days, it takes you 20 minutes to get to work, but sometimes you can make it much faster if you get green lights all the way, and sometimes you take much longer if you get a lot of red lights. Now, imagine your boss is trying to create a very tightly coordinated schedule, and he asks you to predict what time you would be at work. For which scenario would it be easier to estimate a time-of-arrival for your boss? Scenario A might be pretty easy because your time of arrival would be close to the same time every day. Similarly, Scenario B might be more difficult, because your time of arrival will depend on how many red lights you encounter. Instead then, your boss asks you to provide a window of time within which you are very sure (i.e. 95% sure) that you can make it. Of course you!d want to make it as small a window as possible so as to be useful for your boss – but large enough so that you are never considered late. Consider the two scenarios. Would you assign a larger window for Scenario A or Scenario B? Probably Scenario B, because you!ll never know exactly how many traffic lights you might encounter along the way. The Experimental Task
Throughout the study, I am going to show you a screen with 20 "aircraft!, which all move with the same average velocity, and their velocities fluctuate about the same amount across the route. While the aircraft are traversing their routes, you will be able to see a digital read-out of the velocity of each aircraft. As the trial progresses, you will note that, even though the aircraft are all travelling at about the same speed, they each arrive at different times. Sometimes the arrival times will be close together, sometimes they will be more spread out. An elapsed time counter is available at the bottom of the screen.
117
Once all 20 aircraft have completed their routes, a query screen will appear. It will show another aircraft that is about to begin traversing a route. The screen will present the following question: This aircraft!s speed profile is similar in nature to the aircraft you just observed and it
is about to traverse the same length route.
a) Please estimate this aircraft!s time of arrival.
b) Please provide the smallest TOA window that you are 95% sure will contain the
aircraft!s actual Time of Arrival
When you answer these questions, please consider the aircraft that you just observed and remember that you can assume that this aircraft is representative of the aircraft you observed. Also, you should know that there are no 'right' or 'wrong' answers – we are merely interested in your estimate. The first question “Please estimate this aircraft!s time of arrival” is asking you to make your best guess as to when the aircraft will arrive at its destination. For example, if you were planning to meet the aircraft at the destination point, what time would you expect it to arrive? Please provide your answers in seconds, with accuracy to one decimal point. For example type a number like 60.5 in the box on the screen. Press Enter when you are satisfied with your answer. You will not be able to change it once you have pressed enter. For the second question “Please provide the smallest TOA window that you are 95%
certain will contain the aircraft!s actual TOA,” recall that, even though all aircraft were trying to arrive at the destination at the same time, there was some variability around the arrival time. The window you select should represent the smallest possible window of time that you are 95% sure will contain this aircraft!s actual time of arrival. Use the mouse to move the left slider to represent the lower bound of the time window, and the right slider to represent the upper bound of the time window. Again, please provide answers, in seconds, with precision to one decimal point. The actual numbers are shown above the slider scale. Today!s agenda: We will repeat the sequence described above several times throughout the experiment today. The experiment is divided into four blocks, and you can take a short rest break between each block. If you need to stop for any reason before the break, please let me know, but I will not be able to answer any questions once the experiment has started. Whom to contact for more information:
This study is being conducted jointly by NASA Ames Research Center and the University of Toronto. You may seek further information from the following points of contact:
• Becky Hooey, Principal Investigator (650-604-2399) NASA Ames Research Center, Human System Integration Division University of Toronto, Dept. of Mechanical and Industrial Engineering
• David C. Foyle, Co-Investigator (650-604-3053) NASA Ames Research Center, Human System Integration Division,
• Paul Milgram, Co-Investigator (416-978-3662) University of Toronto, Dept. of Mechanical and Industrial Engineering,
If you have read and understood the above information, please sign the attached informed consent form.
118
APPENDIX B2: EXPERIMENT 2 DEMOGRAPHIC QUESTIONNAIRE
1. Name: ________________________________________________________
2. Age: _________________________
3. Gender: Male _________ Female ______________
4. Do you normally wear glasses or contact lenses for any of the following activities? (please
check all that apply)
5. Reading _____ Watching TV _____ Working on computer _____ Driving _____
Other tasks, please specify _____________________________________________
6. Are you wearing either glasses or contact lenses today? No _____ Yes _____
7. Are you currently employed? No _____ Yes _____
If yes, are you employed part-time _____, full-time _____, contract/casual ______
8. What is your job title: __________________________________________________
9. Are you currently a student? No _____ Yes _____
If yes, please state program, institution, and your START date (i.e. A.A / De Anza /
2005)
10. What is the highest level of education that you have completed?
_____ High School / GED
_____Community College (2 year degree) Please specify degree: ___AA or ___AS
_____University (4 year degree) Please specify Degree / University (i.e. B.A. Psych / SJSU)
_____ Graduate School. Please specify Degree / University (i.e., Ph.D. Chemistry /
Stanford)
______Other Education / Diploma / Certificate program (please specify)
11. Do you have (or have you ever had) a Pilot!s license? No _____ Yes _____
If yes, Please specify type (e.g., private / commercial; fixed-wing, rotorcraft)
12. Have you ever taken a course in statistics? No _____ Yes _____
If yes, how many statistics courses have you taken? ____________________
13. Please list the Department / Institution (i.e. Psychology / SJSU) for each course.
119
APPENDIX B3: EXPERIMENT 2 DEBRIEF QUESTIONS
1) Did you use the scratch paper. If so – what did you write down?
2) What strategy did you use to assess the mean velocity?
3) What strategy did you use to assess the velocity window?
4) Were your velocity windows symmetrical or asymmetrical? How did you decide?
5) What strategy did you use to assess TOA
6) What strategy did you use to assess TOA window?
7) Were your TOA windows symmetrical or asymmetrical? How did you decide?
8) Did your strategy change over the course of the experiment?
9) Do you think your windows got smaller with time?
10) Did you find estimating velocity or TOA easier?
120
APPENDIX B4: METHOD FOR DETERMINING ACTUAL SPEED AND TOA SCORES
Actual Speed Scores
To determine the Actual Mean Speed and the Actual Speed Interval, the following process was used to develop a population distribution comprised of 25,000 aircraft, and calculate the mean and 95% CI. The following steps were conducted 9 times, to create actual values for each of 9 combinations of Speed and Variability.
• Randomly drew a series of 10 !segment speeds" for 25,000 aircraft (one speed for each of 10 segments along the route). The segment speeds were drawn from a normal distribution, with a given mean and standard deviation that mapped to the experimental IV conditions.
• Calculated the mean !route speed" for each of the 25,000 aircraft (by averaging each aircraft"s 10 segment speeds)
• Sorted the list of 25,000 !route speeds" in ascending order
Actual Speed – Calculated the mean !route speed" by averaging the mean route speed of the 25,000 aircraft. Actual Speed Interval - Selected the speed from the sorted list of route speeds that represented the 2.5 and the 97.5 percentile. Note, this is equivalent to calculating the 95% CIs using the following formula: CI = M +/- 1.96 * sd / sqrt n Where M = the mean of the normal distribution from which the 250,000 velocities were selected, sd = standard deviation of that normal distribution, and n = 10. Actual TOA Scores
For TOA, the CIs could not be calculated using the CI formula above given that the TOA distribution was non-normal as discussed previously. The following steps were conducted:
• A segment elapsed time was calculated for each of the 10 segment speeds for each of the 25,000 aircraft (as determined above).
• The 10 segment elapsed times of each aircraft were summed to determine the elapsed time for each of the 25,000 aircraft.
• The 25,000 elapsed times were sorted in ascending order
Actual TOA - Calculated the mean of the 25,000 elapsed times
Actual TOA Interval - Selected the elapsed time from the sorted list of times that represented the 2.5 and the 97.5 percentile. Since the TOA distribution is not normally distributed (it is positively skewed), the equation for 95 % CI reported above for Actual Speed Interval was deemed to be inappropriate. Instead, the actual intervals were determined by selecting the 2.5th percentile and the 97.5th percentile from the distribution of route times.
121
APPENDIX C1: EXPERIMENT 3 PROTOCOL
Welcome and thank you for agreeing to participate in this study. The purpose of this study, which will also form a portion of a PhD research project at the University of Toronto, is to understand how people estimate Time-of-Arrival Uncertainty. Please take a few minutes and read this and the attached informed consent form. It describes the study you are about to participate in, and explains your rights as a subject. If you have any questions at any time, please ask the experimenter. Overview
In this study, I am interested in how you estimate time-of-arrival uncertainty. To start with, I!d like you to consider the following scenarios: Scenario A: You just got a new job working 6:00 p.m. to midnight at the community hospital which is 30 miles away on a six-lane freeway that is pretty straight, but hilly. Since you work the night shift you travel against the commuting traffic, so you don!t have much traffic to worry about, and you always try to stick to the speed limit. You set your cruise control for 65 mph and, although you maintain a pretty constant speed, you notice that you go a few mph slower up the hills and a few mph faster down the hills and occasionally you have to turn off cruise control to pass a car on the road. On average, across 100 days, it takes you about 30 minutes to get to work, but of course it is not always exactly 30 minutes since your speed varies slightly because of the hills and traffic. Scenario B: Instead of travelling the freeway, you decide to take the rural road that has a slower speed limit of 30 mph, but is only 20 miles. Since there are a lot of curves you don!t use cruise control. On this road, you encounter a lot of traffic, particularly at 6:00 when most people are returning home from their day jobs, and this means you have to slow down and speed up frequently. On average, across 100 days, it takes you 25 minutes to get to work, but sometimes you can make it much faster if traffic is light, and sometimes it takes you much longer if traffic is heavy. Now, imagine your boss is trying to create a very tightly coordinated schedule, and he asks you to predict what time you would be at work. For which scenario would it be easier to estimate a time-of-arrival for your boss? Scenario A might be pretty easy because your time of arrival would be close to the same time every day. Similarly, Scenario B might be more difficult, because your time of arrival will depend on how much traffic you encounter. Instead then, your boss asks you to provide a window of time within which you are very sure (i.e. 95% sure) that you can make it. That is, if you drove the route 100 times, you would arrive within the time window 95 times. Of course you!d want to make it as small a window as possible so as to be useful for your boss. Consider the two scenarios. Would you assign a larger window for Scenario A or Scenario B? Probably Scenario B, because you!ll never know exactly how much traffic you might encounter along the way. Today!s agenda:
The experiment is divided into three parts, with a short rest break between each block. PART 1: LEARNING SPEED PROFILES
First, you will be shown 20 "vehicles!, each traversing a route that is .5 miles long, which all move with about the same average velocity, and their velocities fluctuate about the same
122
amount across the route. Each vehicle!s speed will change 10 times as it traverses the route (e.g., it will slow down and speed up as if it were encountering varying headwinds). The 20 vehicles in each trial will share the same speed profile – that is, their average velocity is about the same, and their velocities fluctuate about the same amount across the route. There will be four different profiles and they will be identified by colour (Red, Yellow, Blue and Green). Every vehicle with the same colour icon will always share the same speed profile. Your task is to "learn! each of those 4 speed profiles and provide estimates of the average speed and the amount of variability among the speeds. As the trial progresses, you will note that, even though the vehicles are all travelling at about the same speed, each arrives at the destination at a different time. Sometimes the arrival times will be close together, sometimes they will be more spread out. Pay attention to this distribution of arrival times, as it will be important later in the experiment.
PART 2: TIME OF ARRIVAL (TOA) ESTIMATION
In the second part of the experiment, you will be shown only one vehicle at a time. Its speed profile will always be one of the four you saw in Part 1, and will be identified with the same colour code. You will watch the vehicle as it traverses the route to remind you what the speed profile looked like. Part way through the route, the vehicle!s movement will be paused. A query screen will appear that provides the estimated time of arrival (in elapsed seconds) for this vehicle to complete the route, and asks you to estimate the smallest TOA window that you are 95% certain will contain the vehicle!s actual TOA. When answering, you should consider that the vehicle shares the same profile as all of the vehicles with the same colour icon from the first part of the study, and use your knowledge of that speed profile in generating estimates. PART 3: SPEED PROFILE IDENTIFICATION
In the third part of the experiment, you will be shown one vehicle as it traverses the route. Its speed profile colour code will NOT be identified. Rather you will be asked to identify the speed profile from those you observed in PART 1 and 2 of the study, and then estimate the smallest TOA window that you are 95% certain will contain the vehicle!s actual TOA. This study is being conducted jointly by NASA Ames Research Center and the University of Toronto. You may seek further information from the following points of contact for each institution:
• Becky Hooey, Principal Investigator NASA Ames Research Center, Human System Integration Division University of Toronto, Dept. of Mechanical and Industrial Engineering 650-604-2399
• David C. Foyle, Co-Investigator NASA Ames Research Center, Human System Integration Division, 650-604-3053
• Paul Milgram, Co-Investigator University of Toronto, Dept. of Mechanical and Industrial Engineering, 416-978-3662
Detailed instructions will be provided before beginning each part of the experiment. If you have read and understood the above information, please sign the attached informed consent form and tell the experimenter when you are ready to continue.
123
PART I INSTRUCTIONS
You will be shown a screen with 20 !vehicles", each traversing a route that is .5 miles long
(as shown in the picture below), which all share the same speed profile – that is, they move
with about the same average velocity, and their velocities fluctuate about the same amount
across the route. Each vehicle"s speed will change 10 times as it traverses the route (e.g., it
will slow down and speed up as if it were encountering varying headwinds). As the trial
progresses, you will note that, even though the vehicles share the same speed profile, they
each arrive at the destination at different times. Sometimes the arrival times will be close
together, sometimes they will be more spread out. Pay attention to this distribution of arrival
times, as it will be important later in the experiment.
Once all 20 vehicles have completed their routes, a query screen will appear (as shown
below). You will be asked to enter the identification number of the vehicle that finished the
route first, and the vehicle that finished last. You must enter a vehicle number - even if you
aren"t sure, please provide your best guess. Next, you will be shown a new vehicle that is
about to begin traversing a route and will ask you to predict this vehicle"s average velocity
and then to provide the smallest range that you are 95% sure will contain this vehicle"s
124
actual average velocity. When you answer these questions, you can assume that this new
vehicle shares the same speed profile of those vehicles you just observed.
The first question “Please estimate this vehicle!s average velocity” is asking you to make
your best guess as to the average speed that this new vehicle will travel based on your
knowledge of the speed profile that it belongs to. Please provide your answers by typing a
number in the box on the screen – you can enter a number with precision up to one decimal
place (i.e. 52.6). Press “Enter” on the keyboard when you are satisfied with your answer.
For the second question, “Please provide the smallest window that you are 95% certain will
contain the vehicle!s average velocity”, use the mouse to move the left slider to represent
the lower bound of the window, and the right slider to represent the upper bound of the
window. The actual numbers of the window that you have selected are shown below the
slider scale. The window you select should represent the smallest possible range of
velocities that you are 95% sure will contain this vehicle!s average velocity. When you are
satisfied with the window you have selected, click “Continue” with the mouse. Once you
press “Continue” you will receive feedback as to whether each of your answers was correct,
too high, or too low. Answers within 1.0 of the actual answer will be considered correct.
Please note this feedback, and consider it in your subsequent answers. You should strive to
provide the most accurate answers that you can.
125
There will be four different speed profiles – each with a different colour code: Blue, Green,
Yellow, and Red. You will complete five trials of each speed profile. After the fifth trial, you
will be asked to pause and answer a short questionnaire to characterize the profile you just
observed. You will be asked the following three questions for each profile:
1. Based on the average speed, are the vehicles with this profile more likely travelling on a six-lane freeway (i.e. FAST) or a two-lane rural road? (i.e., SLOW) (please circle one) 1 2 3 4 5 6 7
Definitely Definitely two-lane unsure six-lane rural road freeway (i.e., Slow) (i.e., Fast)
2. Based on the speed fluctuations, are the vehicles with this profile more likely traveling with cruise control ON or OFF? (please circle one) 1 2 3 4 5 6 7
Definitely Definitely cruise control unsure cruise control ON OFF 3. Please characterize the time of arrival of vehicles that share this speed profile. Use this to make any notes about the time of arrival that will help you remember later on. Ask the experimenter if you have any questions. We will begin with 4 practice trials, so that you can see one of each of the four speed profiles. Then the experiment will begin.
126
PART 2 INSTRUCTIONS: TIME OF ARRIVAL ESTIMATION
In the second part of the experiment, you will be shown only one vehicle at a time. Its speed
profile will always be one of the four you saw in Part 1, and will be identified with the same
color code (Blue, Green, Red, or Yellow). You will watch the vehicle as it traverses a route
to remind you what the speed profile looked like. Then a new route will be displayed along
with an estimated time of arrival (in elapsed seconds) specific for the distance of route
shown. This estimated time of arrival is based on the average speed, but as you saw in part
1, the actual TOAs will vary. You will be asked to estimate the smallest window that you are
95% certain will contain the vehicle!s actual TOA”. When answering this question, you
should consider that the vehicle shares the same profile as all of the vehicles with
the same color icon from the first part of the study and use your knowledge of the
speed profile to answer the questions. Recall from Part 1, that even though all vehicles
were trying to arrive at the destination at the same time, there was some variability around
the arrival time. The window you select should represent the smallest possible window of
time that you are 95% sure will contain this vehicle!s actual time of arrival -- that is, that
would capture 95 of 100 vehicles. Use the mouse to move the left slider to represent the
lower bound of the time window, and the right slider to represent the upper bound of the time
window. Again, please provide answers, in seconds, with precision to one decimal point.
The actual numbers of the window you select will be shown below the slider scale. You will
not receive any feedback in this portion of the study.
We!ll start with two trials to let you practice, and then we!ll continue with the experiment.
Please ask the experimenter if you have any questions before commencing.
127
PART 3 INSTRUCTIONS: SPEED PROFILE IDENTIFICATION
This task will be identical to Part 2, except, this time the vehicle!s speed profile will not be
identified for you – that is, the vehicle!s color code will not be displayed (all vehicles will be
black). Some vehicles will match one of the speed profiles you experienced in the earlier
portions of the study, but some will not. Your first task will be to determine if it matches one
of the color-coded speed profiles, and if so, which one. Then you will continue to provide the
smallest possible window of time that you are 95% sure will contain this vehicle!s actual time
of arrival. Again, you should use your knowledge of the speed profiles that you learned in
Phase 1 to answer this question.
For each trial, you will be asked to answer the following questions:
1. To which profile is this vehicle most similar?
_____ Red _____ Yellow _____ Green _____ Blue
2. Please rate the degree of similarity between the vehicle you just observed and the profile
you selected above:
Very Low Low Moderate High Very High
1 2 3 4 5
Please ask the experimenter if you have any questions before commencing.
128
APPENDIX C2: EXPERIMENT 3 DEMOGRAPHIC QUESTIONNAIRE
1. Name: ________________________________________________________ 2. Age: _________________________ 3. Gender: Male _________ Female ______________
4. Do you normally wear glasses or contact lenses for any of the following activities?
(please check all that apply) Reading _____ Watching TV _____ Working on the computer _____ Driving _____ Other tasks, please specify _____________________________________________
5. Are you wearing either glasses or contact lenses today? No _____ Yes _____ 6. Are you currently employed? No _____ Yes _____ If yes, are you employed part-time _____, full-time _____, contract or casual hours
______ What is your job title:
________________________________________________________ 7. Are you currently a student? No _____ Yes _____ If yes, please state program, institution, and your START date (e.g. A.A/De Anza/Sept,
2005) ____________________________________________________________________________
8. What is the highest level of education that you have completed? _____ High School / GED _____Community College (2 year degree) Please specify degree: _______AA or
_______AS _____University (4 year degree). Please specify Degree / University (e.g. B.A. Sociology /
SJSU) ________________________________________________________________ ______Graduate School. Please specify Degree / University (i.e Ph.D. Chemistry / Stanford) ________________________________________________________________ ______Other Education / Diploma / Certificate program (please Specify) ________________________________________________________________ 9. Do you have (or have you ever had) a Pilot!s license? No _____ Yes _____
If yes, Please specify type (e.g., private / commercial; fixed-wing, rotorcraft) ___________________________________________________________________
10. Have you ever taken a course in statistics? No _____ Yes _____ If yes, how many statistics courses have you taken? ____________________ Please list the Department / Institution (e.g. Psychology / SJSU) for each course. ____________________________________________________________
129
APPENDIX C3: EXPERIMENT 3 DEBRIEF QUESTIONS
Consider a new speed profile with an average speed of 45 mph. If the vehicles traveled a
route that was 4 miles long at exactly 45 mph, they would arrive in 320 seconds (5 minutes
and 20 seconds). Since their speeds fluctuate due to traffic, 95% of the vehicles arrive
within an 80 second window around the expected TOA.
a) If the vehicles were then to travel a route that was only 2 miles long, the expected TOA
would be 160 seconds (2 minutes and 40 seconds). How big would the TOA window
have to be to include 95% of the vehicles! arrival times?
b) If the vehicles travel an 8 mile route, the expected TOA would be 640 seconds (10
minutes and 40 seconds). How big would the TOA window have to be to include 95% of
the vehicles! arrival times?
50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 210 220 230
Expected TOA Expected TOA = 160
Seconds TOA Window =
Expected TOA Expected TOA = 320
Seconds TOA Window =
210 220 230 240 250 260 270 280 290 300 310 320 330 340 350 360 370 380 390
530 540 550 560 570 580 590 600 610 620 630 640 650 660 670 680 690 700 710
Expected TOA Expected TOA = 640 Seconds TOA Window = _____ seconds
130
1. With respect to the second block of trials – What percent of vehicle TOAs do you
think you captured within your TOA windows?
2. What strategy did you use to assess the TOA windows?
3. Were your TOA windows symmetrical or asymmetrical? How did you decide?
4. Did your strategy change over the course of the experiment? / Do you think your
TOA windows got smaller with time?
5. How did you factor in the distance of the route?
6. Did you factor in knowledge of the speed profile, i.e. other aircraft with the same
color – or just the one aircraft shown in each trial?
7. With respect to the last block of trials – when a vehicle did not match one of the
previously learned speed profiles – how did you determine the TOA windows?
