Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Match-Moving for Area-Based Analysis ofEye Movements in Natural Tasks

Wayne J. Ryan?, Andrew T. Duchowski?, Ellen A. Vincent,and Dina Battisto

?School of Computing, Department of Horticulture, Department of ArchitectureClemson University

ETRA 2010, 22-24 March, Austin, TX

1 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Abstract

Abstract

Analysis of recordings made by a wearable eye tracker is compli-cated by video stream synchronization, pupil coordinate mapping,eye movement analysis, and tracking of dynamic Areas Of Interest(AOIs) within the scene. In this paper a semi-automatic system isdeveloped to help automate these processes. Synchronization isaccomplished via side by side video playback control. A deformableeye template and calibration dot marker allow reliable initializationvia simple drag and drop as well as a user-friendly way to correctthe algorithm when it fails. Specifically, drift may be corrected bynudging the detected pupil center to the appropriate coordinates. Ina case study, the impact of surrogate nature views on physiologicalhealth and perceived well-being is examined via analysis of gazeover images of nature. A match-moving methodology was devel-oped to track AOIs for this particular application but is applicabletoward similar future studies.

2 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Video

3 / 28

match-moving.mp4Media File (video/mp4)

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Motivation

Goal: count fixations in each Area Of Interest (AOI)Problems:

video stream synchronizationgaze position mappingfixation identificationframe-to-frame AOI tracking

4 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Background

Prior natural task application context, e.g.,picking blocks (Ballard et al., 1995)making tea (M. Land et al., 1999)washing hands (Pelz et al., 2000)making PB&J sandwiches (M. F. Land & Hayhoe, 2001)

Manual inspection of video frames was often required(Jacob & Karn, 2003)No particularly easy-to-use system has emerged

5 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Contributions

Our approach consists of:

video synchronization with screen flash and GUIgaze position mapping via 2nd order polynomialfixation identification independent of video streamframe-to-frame AOI tracking via match-moving(Paolini, 2006)

Similar to keyframing and inbetweening (semi-automatic)

6 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Dynamic AOIs

Inspired by Shakes Rotoshape, e.g., rotoscoping Neisserand Becklens (1975) umbrella woman

shake_shape_data 4.0motion_blur 0.000000shutter_timing 0.500000shutter_offset 0.000000num_shapes 1shape_name Shape1parent_nameclosed 1visible 1locked 0tangents 1edge_shape 1num_vertices 5num_key_times 170key_time 952.000000center_x 531.190002center_y 104.557999color_r 1.000000color_g 1.000000color_b 1.000000color_a 1.000000

vertex_data 517.852905 219.235275 517.852905 219.235275 517.852905 . . .

. . .7 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Similarities to and Distinctions from Prior Work

Our work is most similar to Munn et al.s (2008)

Munn et al. advanced trackersanalytical capability in twoimportant ways:

detecting fixations within raweye movement datatracking objects in the scene

Our contributions are distinct from Munn et al.s work:fixation detection is independent from frame rateobject tracking is simpler (2D)

8 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Following Babcock and Pelz (2004) and Li (2006). . .

Pair of safety glasses (AOSafety X-Factor XF503)Nose piece from sunglasses (AOSafety I-Riot 90714)Black polyester braided elastic for wrapping wiresTwo screws to mount scene camera bracketAluminum or brass rod for mounting eye camera

9 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Video Cameras

Video cameras are Camwear Model200 from DejaView (Reich et al., 2004)

camera connected to recorder boxwhich lacks LCDMPEG-4 video recorded to 512 MBSD disk at 30 fpsrecorders do not support videotransfer while recordinglack of LCD prevents verification ofcamera placement

= all processing offlineNo IR illumination used in thisimplementationFor details see Ryan et al. (2008)

10 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Synchronization

Video processing begins with synchronizationCameras may not start recording at the same time(this would be alleviated if cameras could be synchronizedvia hardware or software control, e.g., IEEE 1394 bus)As suggested by Li and Parkhurst (2006), a flash of lightvisible in both videos is used as a markerVideo playback controlled manually for initial alignment

11 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Calibration

Manual template placement facilitates pupil/dot trackingPupil center (x , y) mapped to scene coordinates (sx , sy )via 2nd order polynomial (Morimoto & Mimica, 2005)

sx = a0 + a1x + a2y + a3xy + a4x2 + a5y2

sy = b0 + b1x + b2y + b3xy + b4x2 + b5y2

Unknown parameters ak and bk are computed via leastsquares fitting, e.g., Lancaster and alkauskas (1986)

12 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Dot Tracking

A greedy algorithm is used to track the calibration dotDot modeled as bright center-on surround pixel fieldAssuming an initial dot location, a sum of differences isevaluated over an 88 reference window:

i

j

I(x , y) I(x i , y j), 8 < i , j < 8

repeated over a 55 search field to find brightest pixels

13 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Pupil Tracking

The pupil center (x , y) is found by fitting an ellipse to limbicfeatures detected via variant of Starburst (Li et al., 2005)

ray cast from origin terminates when exiting dark regionpixels with maximum collinear gradient max recorded asfeature pointsassuming frame coherence, feature point search isconstrained according to:

max(O + (P O) : 0.8 < < 1.2)

14 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Fixation Detection

Collected raw gaze points and timestamp x = (x , y , t) areanalyzed to detect fixations in the data streamA position-variance or dispersion-based approach is used

spatial deviation is set to 19 pixelsnumber of samples set to 10

Alternative approach is velocity-based, see Salvucci andGoldberg (2000)The fixation analysis code is freely available

originally made available by LC Technologiesoriginal fixfunc.c can still be found on Andrew R.Freeds eye tracking web page:

http://freedville.com/professional/thesis/readme.htmlC++ interface and implementation ported from C by MikeAshmore available at:

http://andrewd.ces.clemson.edu/courses/cpsc412/fall08

15 / 28

http://www.eyegaze.comhttp://freedville.com/professional/thesis/readme.htmlhttp://andrewd.ces.clemson.edu/courses/cpsc412/fall08

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

AOI Tracking

AOIs tracked by placing trackboxesatop pixel features (e.g., corners)Trackboxes follow features as theymove from frame to frame

Feature tracking is similar to dot tracking with computationreduced by precomputed summed area table (Crow, 1984)

S(x , y) =

i

j

I(i , j), 0 < i < x , 0 < j < y

(each pixel stores the sum of all pixels above and to the left)

A 55 search field is used within an 88 reference windowSum of differences replaced by I(x , y) , where

=(S(A) + S(B)) (S(C) + S(D))

p q16 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

AOI Labeling

t3

t2

t1

x

A

D

GH

E

BC

F

I

For testbed application, 9 AOIs were labeled, A, B, . . ., IAt least three trackboxes were used t1, t2, t3Given a fixation detected at x,

t1 acts as the origin of the reference frametrackboxes t2 and t3 and x translated to the origin (-t1)trackbox t2 defines the rotation angle, = tan1 (t2y /t2x )fixation point x rotated to align with the horizontal x-axistrackbox t3 provides localization (scaling) within 99 gridaxis-aligned and scaled fixation point x assigned label

17 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Experiment: Understanding Health Benefits

Stimulus: four image categories (prospect, refuge,hazard, mixed prospect and refuge) + blank screensMetrics: fixations within AOIs, physiological, andself-reported psychological dataApparatus: 9 53 display wall (subtending 50.2 visualangle at 9.6 viewing distance), hospital bed, bloodpressure and heart rate monitor, eye trackerSubjects: 109 healthy college students, with a smallsubsample (21) participating in the eye tracking portionDesign: gaze analysis performed via repeated measuresdesign (fixations treated as within-subjects fixed factor)

18 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Mean Fixation Duration

600

800

1000

1200

1400

1600

1800

A B C D E F G H I O

Mean F

ixation D

ura

tions (

in m

s; w

ith S

E)

AOI

Fixation Durations vs. AOI

0

500

1000

1500

2000

2500

3000

3500

4000

A B C D E F G H I O

Mean F

ixation D

ura

tions (

in m

s; w

ith S

E)

AOI

Fixation Durations vs. AOI

controlyellow field

treefire

lavender field

10 of 21 recordings discarded due to various factorsANOVA indicates a marginally significant main effect ofAOI on fixation duration (F(9,1069) = 2.08, p < 0.05)Longest durations tend to fall on central AOIs (E and H)Not surprising, particularly in absence of viewing task(Wooding, 2002)

19 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Conclusion

Match-moving approach helps automate analysis of eyemovements w.r.t. dynamic AOIsTechnical contributions addressed

video stream synchronizationpupil detection and mappingeye movement analysistracking of dynamic AOIs

Techniques demonstrated in the evaluation of gaze onimages of nature viewed by participants in a health studyAlthough descriptive statistics over AOIs failed to showsignificance of any particular AOI except the center, themethodology is applicable toward similar future studies

20 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Acknowledgments

This work was supported in part by a US Department ofDefense grant through the Spartanburg Regional HealthSystem.

21 / 28

Introduction Background Approach Testbed Results Conclusion Acknowledgments Q&A

Questions

Thank youComments, Questions?

22 / 28

Appendix References

Selected References I

Babcock, J. S., & Pelz, J. B. (2004). Building a LightweightEyetracking Headgear. In ETRA 04: Proceedings of the2004 Symposium on Eye Tracking Research &Applications (pp. 109114). San Antonio, TX.

Ballard, D. H., Hayhoe, M. M., & Pelz, J. B. (1995). MemoryRepresentations in Natural Tasks. Journal of CognitiveNeuroscience, 7(1), 6680.

Crow, F. C. (1984). Summed-area tables for texture mapping.In Siggraph 84: Proceedings of the 11th annualconference on computer graphics and interactivetechniques (pp. 207212). New York, NY: ACM.

23 / 28

Appendix References

Selected References II

Jacob, R. J. K., & Karn, K. S. (2003). Eye Tracking inHuman-Computer Interaction and Usability Research:Ready to Deliver the Promises. In J. Hyn, R. Radach, &H. Deubel (Eds.), The Minds Eye: Cognitive and AppliedAspects of Eye Movement Research (pp. 573605).Amsterdam, The Netherlands: Elsevier Science.

Lancaster, P., & alkauskas, K. (1986). Curve and SurfaceFitting: An Introduction. San Diego, CA: Academic Press.

Land, M., Mennie, N., & Rusted, J. (1999). The Roles of Visionand Eye Movements in the Control of Activities of DailyLiving. Perception, 28(11), 13071432.

24 / 28

Appendix References

Selected References III

Land, M. F., & Hayhoe, M. (2001). In What Ways Do EyeMovements Contribute to Everyday Activities. VisionResearch, 41(25-26), 35593565. ((Special Issue on EyeMovements and Vision in the Natual World, with mostcontributions to the volume originally presented at theEye Movements and Vision in the Natural Worldsymposium held at the Royal Netherlands Academy ofSciences, Amsterdam, September 2000))

Li, D. (2006). Low-Cost Eye-Tracking for Human ComputerInteraction. Unpublished masters thesis, Iowa StateUniversity, Ames, IA. (Techreport TAMU-88-010)

Li, D., & Parkhurst, D. (2006, 4-5 September). Open-SourceSoftware for Real-Time Visible-Spectrum Eye Tracking. InConference on Communication by Gaze Interaction.Turin, Italy.

25 / 28

Appendix References

Selected References IV

Li, D., Winfield, D., & Parkhurst, D. J. (2005). Starburst: Ahybrid algorithm for video-based eye tracking combiningfeature-based and model-based approaches. In Vision forHuman-Computer Interaction Workshop (in conjunctionwith CVPR).

Morimoto, C. H., & Mimica, M. R. M. (2005, April). Eye GazeTracking Techniques for Interactive Applications.Computer Vision and Image Understanding, 98, 424.

Munn, S. M., Stefano, L., & Pelz, J. B. (2008, July 28-30).Fixation-identification in dynamic scenes: Comparing anautomated algorithm to manual coding. In APGV 08:Proceedings of the 5th Symposium on Applied Perceptionin Graphics and Visualization (pp. 3342). New York, NY:ACM.

26 / 28

Appendix References

Selected References V

Neisser, U., & Becklen, R. (1975). Selective looking: Attendingto visually specified events. Cognitive Psychology, 7,480494.

Paolini, M. (2006). Apple Pro Training Series: Shake 4.Berkeley, CA: Peachpit Press.

Pelz, J. B., Canosa, R., & Babcock, J. (2000). Extended TasksElicit Complex Eye Movement Patterns. In ETRA 00:Proceedings of the 2000 Symposium on Eye TrackingResearch & Applications (pp. 3743). Palm BeachGardens, FL.

Reich, S., Goldberg, L., & Hudek, S. (2004, October 15). DejaView Camwear Model 100. In CARPE04: Proceedings ofthe 1st ACM Workshop on Continuous Archival andRetrieval of Personal Experiences (pp. 110111). NewYork, NY: ACM Press.

27 / 28

Appendix References

Selected References VI

Ryan, W. J., Duchowski, A. T., & Birchfield, S. T. (2008).Limbus/pupil switching for wearable eye tracking undervariable lighting conditions. In Etra 08: Proceedings ofthe 2008 symposium on eye tracking research &applications (pp. 6164). New York, NY: ACM.

Salvucci, D. D., & Goldberg, J. H. (2000). Identifying Fixationsand Saccades in Eye-Tracking Protocols. In ETRA 00:Proceedings of the 2000 Symposium on Eye TrackingResearch & Applications (pp. 7178). Palm BeachGardens, FL.

Wooding, D. (2002). Fixation Maps: Quantifying Eye-MovementTraces. In Proceedings of ETRA 02. New Orleans, LA.

28 / 28

IntroductionVideoMotivation

BackgroundPrior WorkContributionsPrior Work Similarities and Distinctions

ApproachTechnical DevelopmentCamerasSynchronizationCalibrationDot TrackingPupil TrackingFixation Detection

TestbedExperiment

ResultsMean Fixation Duration

ConclusionConclusion

AcknowledgmentsAcknowledgmentsQuestions

Q&AAppendixSelected references

References