Dept. of Library and Information Science, Rutgers University New Brunswick, NJ, USA

Workshop on Inferring Cognitive and Emotional States from Multimodal Measures – MMCogEmS’2011 November 17, 2011

Jacek Gwizdka & Michael J. Cole

!! Overall research goal: infer and predict mental states and context of a person engaged in interactive information search (e.g., Web search)

!! Completed projects: measures derived from eye-gaze patterns !! eye-movement patterns and interaction logs to infer

"! task characteristics "! dynamic user states (such as cognitive load/effort) "! persistent user characteristics (such as domain knowledge)

!! On-going projects: multi-modal measures !! eye-tracking + EEG + GSR !! cognitive load + timing of relevance decisions

3

!! Methodology: Using eye-gaze patterns !! Higher-order patterns: Reading Models !! Measures of cognitive effort in reading

!! Results: !! User study I: journalistic search tasks

"! task characteristics "! cognitive effort

!! User study II: genomics search tasks "! cognitive effort (& learning) "! domain knowledge

!! On-going work

5

!! Eye-tracking research have frequently analyzed eye-gaze position aggregates ('hot spots’) !! spatiotemporal-intensity –

heat maps !! also sequential – scan paths

! Higher-order patterns: reading models & derived measures

!! We have developed a new methodology to analyze eye-gaze patterns: !! Model the reading process to represent (textual) information

acquisition in search

!! Measure the cognitive effort due to (textual) information acquisition

!! Use both to correlate / infer higher-level constructs (task characteristics, user knowledge, etc.)

7

Can be represented as units of reading experience:

((F F F) F (F F F) F F F F (F F F F F F) F) F = fixation

1.! Eye movements are cognitively controlled (Findlay & Gilchrist, 2003)

2.! Eyes fixate until cognitive processing is completed (Rayner, 1998)

Eye gaze pattern analysis is powerful:

!! Eye gaze is only way to acquire (textual) information

!! 1. + 2. ! Direct causal connection between observable (text) information search behavior and user’s mental state

9

!! We implemented the E-Z Reader reading model (Reichle et al., 2006) !! Fixation duration >113 ms – threshold for lexical processing

(Reingold & Rayner, 2006) !! The algorithm distinguishes reading fixation sequences from

isolated fixations, called 'scanning' fixations !! Each lexical fixation is classified to (S,R) that is (Scan,

Reading)

!! Inputs: eye gaze location, duration !! Add fixation to reading sequence if next saccade: !! on the same line of text !! and less than 120 pixels to the right !! or is a regression on the same line of text

11

!! Two states: reading and scanning !! transition probabilities !! each state characterized by the number of lexical fixations

and duration

Scan Read

1-q

p

1-p

q

12

Can be represented as units (Fixations) of reading experience:

(F F F) F (F F F) F F F F (F F F F F F) F Using the reading model : Reading state – R (green); Scanning state – S:

R S R S S S S R S

13

!! Eyes fixate until cognitive processing is completed (Rayner 1998)

!! While reading, words already understood in the parafoveal region are skipped (Reichle, et al., 2006)

!! Eye gaze patterns depend on cognitive processing of information that is being acquired

!! Hypothesis: Analysis of reading fixation patterns reveal some aspects of cognitive effort

14

!! Reading Speed

!! Perceptual Span - Average spacing of fixations

!! Lexical Fixation Duration Excess (LFDE): !! Time needed to acquire meaning above the minimum for

lexical access

!! Fixation Regressions - Number of regression fixations in the reading sequence

text acquired = ----------------- processing time

15

!! Reading speed will be slower for: !! hard to read text (Rayner & Pollatsek, 1989), !! unfamiliar words (Williams & Morris, 2004), !! words used in less frequent senses (Sereno, O’Donnell, &

Rayner, 2006), !! more complex concepts (Morris, 1994)

16

1o (70px) foveal region

Perceptual span is the spacing of fixations

Perceptual span reflects a human limitation on the number and difficulty of concepts that can be processed (e.g. Pollatsek et al. 1986).

17

!! 10-15% of fixations are regressions

!! Reading goal affects reading regressions !! More regressions when: !! greater reader domain expertise, !! conceptually complex & difficult text passages, !! resolution of ambiguous (sense) words

18

!! Greater LFDE indicates less familiar words & greater conceptual complexity

!! LFDE is also correlated with establishing word meaning in context

example

19

20

!! 32 journalism students

!! 4 journalistic tasks (realistic, created by journalism faculty and journalists)

!! Journalism tasks can be about any topic, but few task types.

!! Tasks designed to vary in ways that affect search behavior (Li, 2009)

!! Task difficulty was post-self-rated by participants (7-point Likert scale: ’very easy’ to ’extremely difficult’)

21

!! Complexity - number of steps needed (ex: identify an expert, get contact information)

!! Task Product (factual vs. intellectual, e.g., fact checking vs. production of a document)

!! Named - is actual search target specified?

!! Level - the information object to process (a complete document vs. a document segment)

!! Task Goal - the nature of the task goal (specific vs. amorphous)

!! Note: Copy Editing CPE & Advance Obituary OBI are most dissimilar !! Copy Editing is expected to be easiest, Advance Obituary most difficult

Task Product Level Named Goal Complexity

Background BIC mixed Document No Specific High

Copy Editing CPE factual Segment Yes Specific Low

Interview Preparation INT mixed Document No Mixed A,S Low

Advance Obituary OBI factual Document No Amorphous High

23

!! User task characteristics !! Can we detect task characteristics from eye-gaze patterns ?

!! Cognitive effort !! Do the cognitive effort measures correlate with:

"! task properties expected to contribute to task difficulty? "! the effort needed to complete the task? "! user judgment of task difficulty?

24

25

!! Task effects on transition probabilities S!R & R!S (all subjects & pages)

(Cole, Gwizdka, Liu, Bierig, Belkin & Zhang, ECCE 2010; IwC 2011)

•! For OBI, INT searchers biased to continue reading

•! For CPE to continue scanning

Searchers are adopting different reading strategies for different task types

OBI: advanced obituary INT: interview preparation CPE: copy editing BIC: background information

26

!! For highly attended pages

Total Text Acquired on SERPs and Content per page

Total Text Acquired on SERPs and Content

OBI: advanced obituary INT: interview preparation CPE: copy editing BIC: background information

27

!! For highly attended pages

State Transitions on SERPs per page

State Transitions on Content pages per page

Read ! Scan

Read ! Scan

Scan! Read Scan! Read

OBI: advanced obituary INT: interview preparation CPE: copy editing BIC: background information

28

Measure Related Task Characteristics

Number of state transitions

bias to read Task level and task goal

level: document; goal other than specific (OBI & INT)

bias to scan level: segment and task goal: specific (CPE)

Total text acquired on SERPs Task complexity: More text acquired in BIC and OBI

Text acquired and number of state transitions per page on content pages

Task level: segment and task product: factual (CPE)

Cole, Gwizdka, Liu, Bierig, Belkin & Zhang. (2011). Task and User Effects on Reading Patterns in Information Search Interacting with Computers 23(4), 346 – 362.

Task Product Level Named Goal Complexity

Background BIC mixed Document No Specific High

Copy Editing CPE factual Segment Yes Specific Low

Interview Preparation INT mixed Document No Mixed A,S Low

Advance Obituary OBI factual Document No Amorphous High

For highly attended pages

For all pages

(self-rated after the task)

BIC CPE INT OBI

!!Search effort: task time, pages visited, queries entered

!! Copy Editing (CPE) required the least effort of all tasks

!! Advance Obituary (OBI) required overall most effort (although not the greatest effort of the tasks for every effort measure)

!! For all tasks, for both greater perceived difficulty (self-ratings) and search task effort:

!! higher median LFDE (Kruskal-Wallis chi-squared =125.02, p = 0.03)

!! slower reading speed (ANOVA F-value=5.5 p=0.02)

!! Strongest correlations obtained when considering only the single longest reading sequence on a page

32

!! …

33

!! …

34

!! …

35

!! Cognitive effort measures seem valid

!! Eye gaze pattern cognitive effort measures match with subjective task difficulty

!! Cognitive effort measure results correlate with task characteristics related to task effort

!! e.g. Complex tasks, amorphous goals

36

37 37

!! Words are indicative of concepts and concept features

!! Reading involves: !! knowledge used to understand words,

!! processing concepts expressed in the content, and

!! acquisition of information (and concepts) from the content

!! User knowledge controls interaction during search:

!! selects the words to read, and

!! imposes cognitive processing demands to understand the concepts associated with the words

!! Does user’s knowledge influence information search behavior?

!! Is cognitive effort related to domain knowledge?

39

!! 40 undergraduate and graduate students

!! Rated 409 genetics and genomics MeSH terms

!! 1: No knowledge, ... to 5: Can explain to others

!! Five tasks from 2004 TREC Genomics track

!! Tasks were hard!

!! We use the same methodology to create reading models and calculate cognitive effort measures as in study I

!! Participants’ domain knowledge (PDK) was represented by sum of term ratings !! participants rated MeSH terms !! normalized by a hypothetical expert

•! ki is the term knowledge rating (1-5) •! i ranges over all terms •! ti is 1 if rated or 0 if not •! m number of terms rated by a participant •! The sum is normalized by a hypothetical expert

who rated all terms as 'can explain to others'

These cognitive effort measures were individually correlated with level of domain knowledge.

For all reading sequences:

!! higher domain knowledge ~ lower cognitive effort !! perceptual span (Kruskal-Wallis "2 = 4734.254, p < 2.2e-16)

!! LFDE (Kruskal-Wallis "2 = 5570.103, p < 2.2e-16) !! reading speed (Kruskal-Wallis "2 = 5570.103, p < 2.2e-16)

Similar correlations found for long reading sequences

Long reading sequences might better reflect concept use by participants during information acquisition because of the attention allocated to acquiring that text.

!! For long reading sequences !! We used random forests to construct regression

models from the cognitive effort measures !! Regression results were clustered

(agglomerate hierarchical clustering) !! Random forest model gave us relative importance of

cognitive effort measures as contributing variables in a predictive model !! high importance: reading length (px), LDFE, total duration

of reading sequences (sum of lexical fix), perceptual span !! less important: number of regressions

Random forest model classification errors all participants

only native English speakers

Random forest model cog effort ! domain knowledge correlation with MeSH based domain knowledge

PDKgroups! low! inter! high!low! 8! 0! 0!intermediate! 1! 23! 0!high! 0! 0! 6!

PDKgroups! low! inter! high!low! 3! 0! 0!intermediate! 0! "#! 0!high! 0! 0! $!

!! Ability to detect knowledge levels indicates a possibility of real-time detection of learning of a new material (new domain)

!! Task “phase” analysis: beginning, middle, end !! same random forest model across the three phases !! significant difference : LFDE drops from beg to mid to end

phase, while !! numFix -- not significantly different between phases !! and readingLength increased from middle to end (sig: Kruskal-

Wallis chi^2 = 885.2262, df = 817, p < 0.05)

!! Possible evidence for learning ?

47

!! Eye tracking enables high resolution analysis of searcher’s activity during interactions with information systems

!! There is more beyond eye-gaze locations with timestamps

!! Eye-tracking data: !! can be used to identification of task characteristics !! … cognitive effort !! … domain knowledge

!! High potential for implicit detection of a searcher’s states

!! The reading model methodology and cognitive effort measures are based on many years of empirical research.

!! Eye movements have a direct causal connection to the information acquisition process.

!! This connection is not mediated!

!! Domain independent

!! Document content is not involved

!! Culturally and individually independent

!! Method represents the user's experience of the information acquisition process

!! Real-time modeling of user domain knowledge is possible

!! Processing requirements are low - just need fixation location and duration.

!! Only recent eye movements are needed to calculate cognitive effort.

!! Real-time assessment of cognitive effort

!! Early task session detection of user properties, e.g. domain knowledge and perception of task difficulty

!! Soon enough for a system to make a difference in providing user support

50

51

!! Start with eye-tracking: pupillometry !! info relevance (Oliveria,

Russell, Aula, 2009) !! low-level decision timing

(Einhäuser, et al. 2010)

!! Adds EEG, GSR !! Funded by Google

Research Award

EEG

GSR

Eye tracking

•! Implicit characterization of Information Search Process using physiological devices

•! Can we detect when searchers make information relevance decisions?

Tobii T-60 eye-tracker

Emotiv EPOC wireless EEG headset

52

!! Jacek Gwizdka http://jsg.tel

!! Acknowledgements !! Funding: IMLS Google !! Collaborators:

"! Dr. Nicholas J. Belkin, Dr. Xiangmin Zhang "! Post-Doc: Dr. Ralf Bierig "! Collaborator & PhD student: Michael Cole "! PhD students: Chang Liu, Jingjing Liu "! Master students

53

!! Eye tracking technology is declining in price and in 2-3 years could be part of standard displays. !! Already in luxury cars and semi-trucks (sleep detection) !! Computers with built in eye-tracking

Tobii / Lenovo proof of concept eye-tracking laptop - March 2011