articol tobii 2

8/13/2019 articol tobii 2

1/7

0-7803-9778-9/06/$20.00 2006 IEEE.

Abstract

In usability testing, eye tracking has been used to

gain insight into human behavior that may not be

available through observation or think-aloud

protocols. HCI researchers have suggested that the

eye movement measures of fixation duration and

fixation frequency are related to cognitive

processing ease/difficulty and search

efficiency/inefficiency. This study investigates

whether the connection between these eye movementmeasures and their respective cognitive activities

can be confirmed through user self-reported data.

The findings show that fixation duration and fixation

frequencies are interdependent measures. Analysis

of the self-reported data indicates that although

users responses are limited in what they reveal

about cognitive processes, the responses generally

confirm the eye movement measures.

Keywords: cognitive processing, eye fixation

duration, eye fixation frequency, eye tracking,

usability testing

Introduction

Eye tracking has been suggested as a primary orsecondary data collection method in usability testingbecause, it is argued, eye movements are an indicatorof cognitive processing activity.[1] [2] One of thechallenges of using eye movements in usabilitytesting, however, is relating specific eye movementmeasures to specific cognitive processes.[3]

Exploratory research by Goldberg and Kotval [4]and Cowen [5] suggests specific links exist betweenthe eye movements users exhibit and the ways theycognitively process and search user interfaces andweb pages. A limitation of their research, and ofmost eye-tracking research related to usability, is theuse of eye movements as the sole measure of

cognitive processing. Users perspectives are nottypically sought in eye-tracking studies, even thoughit has been suggested that researchers do so toconfirm their eye movement findings.[6]

In this paper, I take a first step in addressing thisissue by reporting out the preliminary results of astudy designed to test the relationship betweenspecific eye movement measures and cognitiveactivity. In the next section, I discuss the eye

movement measures used in this study: eye fixationduration and eye fixation frequency.

Eye Movements as a Usability Measure

Fixations occur when the eye is focused on aparticular point on the screen and saccades move theeye from one fixation to the next fixation. Thesequential organization of fixations and saccadescreates scanpaths. To limit the scope of this study,only fixation duration and fixation frequency areanalyzed.

Fixation DurationFixation duration is thought to correspond tocognitive processing ease or difficulty. Longfixations indicate that a person is having difficultycognitively processing the object at which his/hergaze is directed. In scene perception research, out-of-context objects generate longer fixations thancontextual objects.[7] [8] People would most likelyfixate longer on a cow placed in a downtown setting,for instance, than a person in the same setting

because of the incongruous relationship between theobject and its context.

Goldberg and Kotval [4] have extended sceneperception research to user interface design. Insteadof using an out-of-context object to study therelationship between fixation duration and cognitive

Is Eye Tracking the Next Step in Usability Testing?

Lynne CookeUniversity of North [email protected]


2/7

processing, they used toolbar icons, which representa stimulus more appropriate to usability. Toolbaricons that were not logically grouped togethergenerated slightly longer fixations than toolbar iconsthat were intuitively grouped together. From thisfinding, Goldberg and Kotval concluded that

[l]onger fixations imply the user is spending moretime interpreting or relating the componentrepresentation in the interface to the internalizedrepresentation.[4, p. 643]

As the next step in exploring fixation duration as ameasure of usability, Cowen [5] compared 17participants eye movements across four web sites ofvarying spatial density. Density was quantitativelyassessed by a clutter index.[5, p. 20] In addition toidentifying the number of elements (such as anavigation menu) onscreen, the clutter index also

permitted Cowen [5] to measure the number ofelements relative to the amount of space on the page.Pages that had more elements in smaller areas wereranked higher on the clutter index. Notcoincidentally, Cowen [5] found the web page thatranked the highest on the clutter index generated thelongest fixations. Although her findings werepreliminary, they indicate users find pages withdensely clustered items more difficult to cognitivelyprocess than pages with more spacious layouts.

Fixation FrequencyFixation frequency (the total number of eye fixationson a screen) is thought to correspond to searchefficiency; the lower the number of fixations on adisplay, the more efficient the search. According tovisual search research, specific elements caninfluence search; color, size, orientation, and motionwill typically generate high numbers of fixations.[9][10] For example, an animated advertisement on aweb page may increase the fixation count on a pagebecause the motion temporarily diverts a usersattention during the search process.

To continue with Goldberg and Kotvals [4] study ofuser interfaces, they found that the unintuitivetoolbar icon grouping generated 17% more fixationsthan the intuitive icon groupings. Since users weresearching for a specific icon, Goldberg and Kotval[4] speculated that the unintuitive icon groupinginterfered with the efficiency of the search process.Users could not immediately isolate the target icon

and therefore made several unnecessary fixationsbefore completing the search process. The intuitiveicon grouping, by contrast, generated fewer fixationsbecause it allowed users to zoom in on the correctcomponent more efficiently.[4 p. 643]

Cowen [5] also collected fixation frequency data inher study of spatial density. Again, the web pagewith the highest clutter index rating also generatedthe highest number of fixations. This suggests thatusers not only found the page difficult to cognitivelyprocess, but they also had difficulty finding theirsearch target. Since users performed two non-sequential tasks on each of the four web pages, itwas possible to see whether repeated viewing of aweb page influenced eye movement. When usersreturned to a page to perform the second task thenumber of fixations decreased in all but one case.

This finding indicates that users search efficiencyincreased, most likely because they did not have toreorient themselves to the page.

Research Questions

The relationships of eye movements to cognitiveprocesses that Goldberg and Kotval [4] and Cowen[5] put forth are based on a single measure: eyemovement. The present study addresses the gapbetween the objective measure of user eyemovement and the subjective measure of user

accounts of their experience. The present study asksthe following research questions:

Can the interpretation of fixation duration as ameasure of cognitive processing ease ordifficulty be confirmed by users?

Can the interpretation of fixation frequency as ameasure of search efficiency or inefficiency beconfirmed by users?

Study Design

The study uses a repeated measure, within-subjectsapproach. Each participant completed three tasks onthree web pages. This resulted in 78 web pages foranalysis. Eye fixation duration and eye fixationfrequency were treated as separate eye movementmeasures. A survey was used to collect participantsfeedback about their ease or difficulty of completing


3/7

0-7803-9778-9/06/$20.00 2006 IEEE.

the tasks. Data was then analyzed on web page andparticipant levels.

ParticipantsTwenty-six people participated in the study, 21

women and 5 men. Participants ages ranged from 19to 36, and all but one person was an undergraduateor graduate student at the University of Washington.Participants were recruited by e-mail; only peoplewho did not wear glasses or contacts were eligiblefor the study due to limitations of the eye tracker.

All participants were regular computer and Internetusers. Their estimated time spent on the web perweek varied. Two participants spent more than 40hours a week, eight participants spent 21-40 hours aweek, six participants spent 11-20 hours a week, and

ten participants spent 1-10 hours a week. The studywas approved by the University of WashingtonsHuman Subjects Review Board and the University ofNorth Texass Institutional Review Board.Participants signed Informed Consent Forms foreach university.

Equipment

An ERICA eye tracker and GazeTracker softwarewere used to collect eye movement data. ERICA is aremote eye-tracker. Eye movements were sampled at

30 mhz. A chin rest was used to minimize headmovement and a 15 flat panel monitor set at aresolution of 1024 x 768 pixels was used to displaythe web pages.

StimulusThe homepages from three web sites were used inthis study: Argonne National Laboratory Washington D.C. Commission on the Arts and

Humanities

Colorado State GovernmentFrom this point forward, these web pages will bereferred to as Argonne, D.C. Arts, and Colorado.

The study included three web pages because nobenchmarks exist about what constitutes high or lowfixation frequencies or long or short fixation

durations. Therefore, the study uses three web sitesto provide a comparable measure for these eyemovements.

These particular web pages were selected because itwas unlikely that participants regularly visited the

three sites. Participant familiarity with the web siteswould have likely influenced their viewing behavior.None of the participants reported visiting any of theweb sites within the last six months. Each participantwas randomly assigned to 1 of 3 web page viewingorders to minimize interaction effects.

The three web sites were also chosen because theyhave similar layouts, but different presentationstyles. The web pages were altered slightly toaccommodate the eye-tracker. Pages were shortenedto eliminate scrolling and in all cases, this required

elimination of the bottom navigation menus.Dynamic html elements (such as drop-down menusand rollover effects) were also removed.

TasksParticipants were asked to find the answers to thefollowing questions:

Argonne: What student internships doesArgonne National Laboratory offer?

D.C. Arts: When is the International FilmFestival?

Colorado: What is Colorados state bird?Wording from the task did not appear in the linknames, as this would have influenced participantssearch behavior. The task ended when participantsselected a link or when participants input a term inthe search box and hit enter.

Procedure

Before beginning the first task, participants received

a standardized verbal orientation to the test and werethen calibrated to the eye-tracker. Followingcalibration, participants were directed to a web pagewith the first task and instructed to click on the linkfor the web page once they had finished reading theprompt. This was repeated two more times for thetwo remaining tasks.


4/7

Upon completion of the last task, participants weregiven a survey to fill out. In the survey, they wereasked to rate the ease or difficulty of completingeach task and then to explain their reason for therating. Following the survey, a retrospective think-aloud protocol session took place; however, the

results from these sessions address a differentresearch question, and therefore, they are notdiscussed. At the end of the study, each participantreceived a small honorarium. Each test session lasted45 minutes per participant.

Results

The eye movement data and task completion timesare presented first, followed by the survey data.

Eye Movement DataTable 1:Mean fixation duration, fixation frequency,task completion times, and standard deviations

Web

Page

Mean

Fixation

Duration

Mean

Fixation

Frequency

Mean Task

Completion

Time

Argonne .821 sec /6.14 SD

2.07 / 15.3SD

14.7 sec /10.5 SD

Colorado .903 sec /5.28 SD

22.5 / 12.8

SD15.3 sec /7.01 SD

D.C. Arts 1.17 sec /6.75 SD

25.8 / 16.1SD

16.0 sec /8.57 SD

From the eye movement data alone, as shown inTable 1, the D.C. Arts web page, with a meanfixation duration of 1.17 seconds, appears to be themost difficult of the three web pages for participantsto cognitively process. The Argonne web page, witha mean fixation duration of .821 seconds, on theother hand, appears to be the easiest web page for

participants to cognitively process. The difference inmean fixation duration times across web pages wasnot significant, F(2, 75) = 2.39,p= .098. (For thisand all other tests, statistical significance wasassumed ifp


5/7

0-7803-9778-9/06/$20.00 2006 IEEE.

Figure 2. An efficient search on the Argonne webpage.

Survey Data

Survey data were collected after participantscompleted their final task. For each of the webpages, participants were asked to rate the ease ordifficulty of completing each task and to explain thereasons for their answers. This measure was used asa self-reported indicator of cognitive processingease/difficulty and search efficiency/inefficiency.

Twenty-one of the 26 participants rated Argonne as aweb page where it was Very easy or Somewhateasy for them to complete their task. Fifteen

participants and eight participants rated Coloradoand D.C. Arts web pages as Very easy orSomewhat easy, respectively, for task completion.In terms of task completion difficulty, 15 of the 26participants rated the D.C. Arts web page asSomewhat difficult. Five participants rated theColorado web page as Somewhat or VeryDifficult, and only one participant rated theArgonne web page as Somewhat Difficult. Fromthe ratings alone, it appears that task completion onthe Argonne page was easier than on the Coloradoand D.C. Arts web pages.

In order to understand better whether a relationshipexisted between eye movements and cognitiveprocesses, data was also analyzed on participant andpage levels.

Participants survey responses were coded forindicators of cognitive processing ease/difficulty and

search efficiency/inefficiency and then matchedagainst their mean eye fixation durations and eyefixation frequencies. The mean for each eyemovement measure was used as the cut-off point todefine cognitive processing ease/difficulty andsearch efficiency/inefficiency. Times/numbers that

fell below the mean were judged easier toprocess/search, while times that fell above the meanwere judged more difficult to process/search.

From the 78 responses, the sample was reduced to 38because only these statements contain informationthat could be interpreted as indicators of cognitiveprocessing and/or search efficiency.From these 38 statements, 32 statements (84%)confirmed the eye movement measures. In the othersix cases, the statements contradicted the eyemovement measure. Some participants who had

higher than average fixation frequencies, forexample, reported that the task was Very easy tocomplete because the page layout made it easy forthem to find the appropriate link. The reverse wasalso true.

Discussion

From comparison of the aggregated eye movementand survey data, the measures of eye fixationduration and eye fixation frequencies are compatiblewith the measure of perceived task difficulty.

Argonne, the page that was presumably the easiestfor participants to cognitively process and search,also received the highest number of Very easy andEasy ratings. Similarly, D.C. Arts, the page thatwas presumably the most difficult for people tocognitively process and search also received thehighest number of Somewhat difficult ratings. Themean task completion times also appear to becompatible with the survey and eye movement data.

On participant and page levels, determining directrelationship between eye movements and subjective

responses proved to be more difficult. Usually,participants written explanations for their ratingsdid not contain enough information that could beinterpreted as evidence of cognitive processingease/difficulty or search efficiency/inefficiency. Thefollowing statement made by a participant regardingthe ease with which he completed the task of findinginformation about student internships on the


6/7

Argonne web page was typical of such a response.The task was easy [b]ecause there was a link thatsaid Job Opportunities with the lab.

It was also difficult to separate the measures ofsearch efficiency and cognitive processing from

participants statements. For example, oneparticipant who had difficulty completing the task offinding information about the International FilmFestival on the D.C. Arts page remarked that thelinks in the left navigation menu were tough toread because of the thin print and underlining.Everything was too close together. Her responsesuggests that she had difficulty both cognitivelyprocessing and efficiently searching the pagebecause of the underlined text and tight spacing.Similarly, another participant who found it easy tocomplete the task on the D.C. Arts page said, News

and events are always in the middle [of the page] andthats where it [the link] was. This participantsresponse also indicates cognitive processing ease andsearch efficiency. Because the link placementmatched the participants expectations, it enabled herto process the page easily and to find the targetinformation quickly.

Distinguishing between search efficiency andcognitive processing in participant responses mayhave been challenging because the two measuresappear to be linked to one another. That is, if a

persons fixation duration was above the fixationduration mean, it was highly likely that their fixationfrequency was also be above the fixation frequencymean and vice versa. For the present study, such arelationship was found in 87% of the participants.Although Goldberg and Kotval [4] reported fixationduration and fixation frequency as differentmeasures that are related to different cognitiveactivities, it appears that on a web page level, searchefficiency/inefficiency and cognitive processingease/difficulty are connected. Not surprisingly, theseeye movement measures are also related to task

completion time. Ninety-two percent of participantseye movements also matched their task completiontimes. Participants whose fixation durations timesand fixation frequencies were above the mean alsohad task completion times that were higher than themean. The opposite was also true.

Conclusion

Although participants were sometimes unable toarticulate their reasons for their task ease/difficultyratings, the information they provided supported theinterpretation of eye fixation duration as a measure

of cognitive processing ease/difficulty. Participantsinformation also supported the interpretation of eyefixation frequency as a measure of both cognitiveprocessing ease/difficulty and search in/efficiency.

Is eye-tracking the next step in usability testing? Itcould be. Currently, eye tracking can tell us wherebut not why users look for information on web pagesor computer interfaces. To gain insight intoconnections between eye movements and cognitiveprocessing, eye tracking and other measures of userbehavior, such as retrospective think aloud protocol,

should be studied together. Future studies of thistype could facilitate the adoption of eye tracking as ausability evaluation method by both researchers andpractitioners.

References

[1] M. A. Just, and P. A. Carpenter, Eye fixationsand cognitive processes, Cognitive Psychology,vol.8, pp. 441-480, 1976.

[2] J. H. Goldberg and A. M. Wichansky, Eye

tracking in usability evaluation: A practitionersguide, in The Minds Eye: Cognitive and AppliedAspects of Eye Movements. J. Hyona, R. Radach, andH. Duebel, Eds. Amsterdam, The Netherlands:Elsevier, 2003, pp. 493-516.

[3] R. J. K. Jacob and K. S. Karn, Eye tracking inhuman-computer interaction and usability research:Ready to deliver the promises, in The Minds Eye:Cognitive and Applied Aspects of Eye Movements. J.Hyona, R. Radach, and H. Duebel, Eds. Amsterdam,The Netherlands: Elsevier, 2003, pp. 573-605.

[4] J. H. Goldberg and X. P. Kotval, Computerinterface evaluation using eye movements: Methodsand constructs,International Journal of IndustrialErgonomics, vol. 24, pp. 631-645, 1999.

[5] L. Cowen (2001).An Eye Movement Analysis ofWeb-Page Usability.Unpublished masters thesis.


7/7

0-7803-9778-9/06/$20.00 2006 IEEE.

Lancaster University. Available:http://www.cs.ucl.ac.uk/staff/J.McCarthy/pdf/library/eyetrack/laura_cowen.pdf

[6] B. Pan, L. Granka, H. A. Hembrooke, M. K.Feusner, G. K. Gay, and J. K. Newman, The

determinants of Web page viewing behavior: An eyetracking study, in Proc. of the Eye TrackingResearch & Applications Symposium, New York,NY: ACM. 2004, pp. 147-154.

[7] G. R. Loftus and N. H. Mackworth, Cognitivedeterminants of fixation location during pictureviewing,Journal of Experimental Psychology:Human Perception and Performance,vol. 4, pp.565-572, 1978.

[8] J. M. Henderson, P. A. Weeks, and A.

Hollingworth, The effects of semantic consistencyon eye movements during complex scene viewing,Journal of Experimental Psychology: Human

Perception and Performance, vol. 25, pp. 210-228,1999.

[9] T. J. Doll, Preattentive processing in visualsearch, in Proc. of the Human Factors andErgonomics Society 37th Annual Meeting.SantaMonica, CA: HFES. 1993, pp. 1291-1249.

[10] J. M. Wolfe, K. R. Cave, and S. L. Franzel,

Guided search: An alternative to the featureintegration model for visual search, The Journal ofExperimental Psychology: Human Perception and

Performance,vol. 15, issue 3, pp. 419-433, 1989.

Acknowledgements

I would like thank Judith Ramey and ElisabethCuddihy of the Technical CommunicationDepartment at the University of Washington for theirhelp with this study and for the use of LUTE. Iwould also like to thank Will Strobel for his web

work and Joe Downing for his comments on a draftof this paper. This study was funded through aUniversity of North Texas Research OpportunitiesGrant.

About the Author

Lynne Cooke is an Assistant Professor of TechnicalCommunication at the University of North Texas.She teaches courses about usability test methods,print and online documentation, and visual

communication. Her eye-tracking research addressesuser search and navigation of web pages.

Documents

articol tobii 2