Getting a Measure of Satisfaction from Eyetracking in Practice

Workshop 24

Organisers:Tony Renshaw, Leeds Metropolitan University, UK

Natalie Webb, Amberlight Partners, London

Janet Finlay, Leeds Metropolitan University, UK

Aims:

• Initially– Define best practice in eye tracking

• Suggest solutions to problems• Highlight unanswered questions• Explore both scientific and commercial use

– Explore how to measure satisfaction with eyetracking

• Adjusted through the day!

Format of Workshop:

• Presentations – 3 minutes per person identifying 3 key issues from position papers

• Participants note issues and questions of interest from presentations on post-its

• Participants organise their questions under themes for discussion and choose groups

• Break out groups – 3 AM, 3 PM• Presentations from break out groups:

– Areas of agreement– Unresolved issues– Future Work

• Summary – key take away points

Breakout Sessions

6 themes:1. When should we use/not use eyetracking?2. What are the best eye tracking

methodologies?3. How do we make eye tracking data analysis

manageable?4. How can we analyse gaze paths? 5. How can we tie eye tracking metrics into

measuring user experience?6. How do we deal with dynamic stimuli?

ParticipantsRepresenting 6 countries and 3 continentsRepresenting both research and commercial users

• Sune Alstrup• Duncan Brumby• Edward Cutrell• Andrew Duchowski• Laura Granka• Ying-Hua Guan• John Hansen• Keith Karn

• Craig Lindley• Janet Read• Jens Riegelsberger• Alain Robillard-Bastien• Kerry Rodden• Anthony Santella• Charlotte Sennersten• Katerina Tzanidou

1. When should we use/not use eye tracking?• Areas of agreement

– Use when appropriate but not always!• Context and purpose of project is important – and very different

between commercial and research• Should always consider complementary techniques• Should recognise and work within the limitations of the technologies

• Unresolved issues– How can we best combine eye tracking with other methods

• How much influence and interference?– Replay gaze trails → post hoc rationalisation?– Concurrent think aloud → alter eye movement behaviour?

– Dealing with limitations from mind-eye hypothesis– Common reporting standards needed e.g. providing full

information about calibration and people who could not be tracked• Future work

– Investigate individual and population differences in eye gaze– Better understand the influence of sampling on eye tracking

studies– Find ways to combine physiological and eye tracking data

2. What are the best eye tracking methodologies?• Areas of agreement

– Must have a list of methods in order to provide “best practices”

– Cannot make sense of data external to intentions• Very different intentions for different research goals (usability

vs cognitive models)

• Unresolved issues/Future work– Refining eye tracking methodologies– Further detailing the best usage of retrospective

reporting vs. concurrent think aloud

3. How do we make eye tracking data analysis manageable?

• Areas of agreement– Plan carefully with a clear view of purpose, problem being addressed and

tasks– Choose appropriate numbers of participants, task sizes and session

lengths– Select appropriate fixation parameters (duration and area)

• Unresolved Issues– Analysis software provided with eye tracking hardware not adequate –

need better facilities for• Filtering• Summation• Vizualization

– Improving ways to combine data with other protocols; e.g. verbal– Studies involving different applications have different analysis

requirements e.g. games, learning systems• Future Work

– Can borrow analysis algorithms from other areas e.g. bioinformatics, machine learning

– Need Open Source analysis tools– Need more publications focusing on methodology and analysis techniques

4. How can we analyse gaze paths?• Areas of interest

– Emerging standard• E.g. String Edit Distance

• Unresolved issues– Dealing with

• Strings of varying lengths• Comparison of multiple strings• Visual presentation of analysis• Very long sequences

– Integration of data from other sources– Annotation of edited strings with comments

• Future Work– All of the above!

5. How can we tie eye tracking metrics into measuring user experience?• Areas of agreement/Unresolved Issues

– Definition of satisfaction is problematic – depends on domain – engagement or immersion may be more useful

– Currently lack of models connecting eye tracking metrics and higher level measures of user experience

– Would like to be able to correlate particular behaviours with measures

– Might be more helpful to approach this from behavioural rather than cognitive perspective

• Future work– Need more models linking eye behaviour to user experience e.g.

• Novice vs. expert behaviour • Certain scan paths indicating a certain depth of engagement (e.g. in

games)

– Using eye movement behaviour to inform design through “design patterns”

6. How do we deal with dynamic stimuli?• Areas of agreement

– Very little known!• Unresolved issues

– Large variety of stimuli formats e.g. games, web applications, java, flash

– Problems of tracking moving AOI (areas of interest), definition of dynamic AOI

– Labelling of data is labour intensive– What granularity of information is required?– How can scan paths be determined in this type of stimuli?

• Further work– Formulate appropriate cognitive and visual influence theories– Explore role of peripheral vision in dynamic environments– Explore influence of auditory information– Develop suitable strategies for determining AOI online or post

hoc• Extract stimuli information directly from the application and integrate

with eye tracking data

Outcomes:• Aim 1: “Best practice” is premature – we

first need a better understanding of possibilities

• Aim 2: “Satisfaction” seen as a problematic construct – we need to develop more meaningful ways of interpreting eye movement data within specific contexts e.g. identifying common patterns of behaviour for specific scenarios

Wish list:

• High level models and metrics• Better analysis algorithms• More flexible and powerful analysis tools• Methodologies for combining eye tracking

with other usability approaches• Better understanding of constraints and

parameters for eye tracking studies in specific contexts

• More research!

If you are interested in eye tracking either as a researcher

or for commercial use, we would like to talk to you!

Please leave us your card in the tray provided and/or take one

of ours.

Thanks for your interest.