Measuring users' experience - or, the memory of them?

Measuring users’ experiences

Evangelos Karapanos

Newcastle, 8 March 2011

or, the memory of them?

Trajectory reminders EmoSnaps Footprint tracker

• 20 faculty– 14 countries, 8 languages

• Areas:– 11 CS, 2 physics/electronics, 2 psychology, 2 architecture, 2 design, 2 art, 2 other

Leonel Nóbrega

Ian Oakley Luis Gomes

ValenHna Nisi

Paulo Sampaio

Gonçalo GouveiaLaura Rodríguez

Vassilis Kostakos

Jos van Leeuwen

Nuno Nunes

Larry ConstanHne

Eduardo Fermé

Néstor Catano

Monchu Chen

Pedro Campos

Barbara Pizzileo

Ron Salden

Evangelos Karapanos

David Aveiro Luis Gomes Yoram Chisik

MSc HCI & Entertainment Technology

Industry Involvement

BSc Physics / microelectronics, U Patras, Greece (2004)Thesis: Model based design and evaluation of walk-up-and-use interfaces (HCI Group, ECE department)

My background

MSc HCI / UCL Interaction Centre, UK (2005)Thesis: User acceptance of nomadic user interfaces (Philips Research, Eindhoven)

PhD HCI / TU Eindhoven, NL (2010)Title: Quantifying diversity in user experience

Socially translucent eco-feedback technologies

!

How do eco-feedback technologies: a) raise mutual awareness of family members’ consumption behaviorsb) induce feelings of accountability on individuals regarding their consumption behaviors.

Citizen participation on the goHow can we motivate citizen participation through mobile technologies?•Public transit: The role of psychological empowerment: self-efficacy, sense of community, and causal importance•Inclucity: The role of visual and location cues on users’ ability to reconstruct the context, and form an empathic understanding of the experience of disabled individuals

Location-aware narratives: Does locality matter?

Some of my current work (that I will not discuss today)

Does the coupling between physical and virtual space result to increased immersion in the narrative world?

User experience over timeWhat makes for positive experiences in the long run?

iScale - longitudinal data through memoryCan an online survey tool assist users in recollecting their experiences with a product?

Technology Assisted Reconstruction

Outline

Can mobile sensors assist participants in reconstructing their daily experiences and whereabouts?

48% of returned products are not attributed to a violation of product specifications

Soft Reliability

problems rooted early in (concept) design phase

failure to truly incorporate it in one’s life

How do users form overall evalua/ve judgments about interac/ve products?

Hassenzahl, 2004

An exploratory studyUser experience over Hme

Karapanos, E., Hassenzahl, M., & Martens, J.-B. (2008). User experience over time. CHI ’08 extended abstracts on Human factors in computing systems (pp. 3561-3566). Florence, Italy: ACM.

Items close together are highly correlated. Lines represent clusters.

An exploratory studyUser experience over Hme

Karapanos, E., Hassenzahl, M., & Martens, J.-B. (2008). User experience over time. CHI ’08 extended abstracts on Human factors in computing systems (pp. 3561-3566). Florence, Italy: ACM.

What makes for positive experiences in the long run?Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. (2009). User experience over time: an initial framework. In CHI '09: Proceedings of the 27th international conference on Human factors in computing systems (pp. 729-738). New York, NY, USA: ACM.

Now think of the three experiences that were for you personally most sa.sfying or unsa.sfying experiences of today. Please, use your own feeling or a defini.on of what “sa.sfying” and “unsa.sfying experience” means. Take a couple of minutes to be sure to come up with three most crucial experiences; you may also want to write them down for yourself. We want you to be open as to which experiences to report.

Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. (2009). User experience over time: an initial framework. In CHI '09: Proceedings of the 27th international conference on Human factors in computing systems (pp. 729-738). New York, NY, USA: ACM.

Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. (2009). User experience over time: an initial framework. In CHI '09: Proceedings of the 27th international conference on Human factors in computing systems (pp. 729-738). New York, NY, USA: ACM.

“Longitudinal” paradigms in HCI

Cross-‐sec8onal Repeated sampling

Longitudinal Retrospec8ve

Karapanos, E., Martens, J.-B., Hassenzahl, M. (2009) Reconstructing Experiences through Sketching. Arxiv preprint, arXiv:0912.5343.

http://ekarapanos.com/iScale.pdf

http://ekarapanos.com/iScale.pdf

Remembering is an act of reconstruc.on rather than reproduc.on

Barlea (1932)

How do we recall experiences?

• Validity? i.e. do memories reflect what we really experienced?

• Reliability? i.e. in a second trial, will we recall the same experiences?

Can we assist people in recalling - more reliably - their experiences with a product?

How do we recall emotional experiences?

Two schools of thought- The Constructive approach- The Value-Account approach

The Constructive approachThe emotional experience can neither be stored nor retrieved. It is reconstructed from recalled contextual details

produce the exact past event, but instead, every attempt to recall results in a new, often altered representation of the event. Bartlett asked participants to recall an unfamiliar story that they were told 20 hours before. Recalled stories differed from the original one in missing details, altering the order and importance of events, or in applying rationalizations and interpretations to the original story. Stories were further distorted through repeated reconstruction.The notion that remembering is an act of reconstruction instead of mere reproduction has received wide support. At the heart of reconstruction lies the distinction between episodic and semantic memory [69]. While episodic memory “is specific to a particular event from the past, semantic memory is not tied to any particular event but rather consists of certain generalizations (i.e. beliefs) that are rarely updated” (p. 935). These two types of memory serve different needs such as learning new information quickly - a capacity of episodic memory - or developing relatively stable expectations about the world - a capacity of semantic memory [63]. Reconstruction happens through the retrieval of cues from episodic memory. In the absence of contextual cues in episodic memory, beliefs found in semantic memory may be used to reconstruct the past, resulting in distortions such as the ones found in Barlett’s study. Thus, overall, the accuracy of one’s remembered events lies in the degree to which contextual cues are still present in the person’s episodic memory.

But, how do we reconstruct emotional experiences that contain not only contextual details of the experienced event, but also value-charged information such as emotions or overall evaluative judgments on the event?

Robinson and Clore [63] argue that “emotional experience can neither be stored nor retrieved” (p. 935), but can only be reconstructed on the basis of recalled contextual cues. They propose an accessibility model that distinguishes between four types of knowledge used to construct an emotion (see figure 1). First, experiential knowledge is used when an emotion is constructed online, i.e. as the experience takes place. It is considered the only “authentic” source of information, yet it is accessible for only a small fraction of time, about 3 seconds [37]. Self-reports tapping into experiential information, such as ESM, are considered the “gold standard” of experience measurement [36] as they avoid biases caused by retrospection.

When experiential knowledge is inaccessible, people will resort to episodic information, i.e. they will recall contextual cues from episodic memory and infer their emotions on the basis of these contextual cues. For instance, in recalling one’s emotions while experiencing a roller coaster, one might recall having his hands raised and screaming, and on this basis to infer an aroused experience. The accuracy with which people may recall their emotions will thus depend on the richness of recalled episodic

information. Daniel Kahneman and colleagues [36] relied on this proposition to develop the day reconstruction method. DRM asks participants to list their daily activities as a continuous list of episodes [36]. This is assumed to form stronger temporal and semantic links across the distinct experiences, thereby minimizing bias from overly relying on semantic information, which is detached from the actual experience. It has been shown that DRM provides a reasonably good approximation to experience sampling data, both in between-subjects [36] and within-subject [20] analyses. When episodic memories become inaccessible, people will shift to semantic memory. People will first access situation-specific beliefs, i.e. “a belief about the emotions that are likely to be elicited in a particular type of situation” [63]. If event-specific beliefs are inaccessible, e.g. due to rarity of the event, people will access identity-related beliefs, i.e. “beliefs about their emotions in general”. Such information is detached from the actual experience and is associated with retrospection biases found in global reports of experience such as online surveys and questionnaires used at the end of empirical studies [65].

A REVIEW OF EXPERIENCE SAMPLING AND DAY RECONSTRUCTION STUDIESThe experience sampling method has been popular in HCI for over a decade. Yet, to our knowledge, no attempt has been made to review the use of the method by HCI researchers. In the following section we attempt a review of empirical studies, published within HCI, that employ either the experience sampling or the day reconstruction method. The questions we will attempt to adress are: How are these methods used within HCI? What are the benefits these methods offer in addressing particular HCI problems? And, what drawbacks are experienced in practice?

Experience samplingAs ES studies have been published across a wide range of HCI venues we decided that narrowing our search down to

2

Figure 1. Four sources of information in emotional self-report according to Robinson and Clore [63]. Figure adapted from [63].

Experiential Knowledge

Online emotion, e.g.Experience Sampling

Episodic memory

Situation-specific belief

Identity-related belief

Retrospective, e.g.Day Reconstruction

Exit questionnaires

Type of Knowledge Source of information Type of Self-Report

Exit questionnaires

Semantic

Semantic

Episodic

Episodic

Robinson & Clore (2002)

Design principles1. Feed-forward sketching, as each recalled event

will cue more events, eventually resulting to richer episodic memories from which to infer emotions

2. Concurrency between sketching and reporting, as reporting will positively contribute towards the recall of episodic cues

The Value-Account approach

(Betsch et al., 2001)

Design principles1. Top-down sketching (i.e., split completed line in

parts), as participants are expected to have direct access to this value-charged information

2. Non-concurrency between sketching and reporting, as reporting (i.e., thinking about concrete episodic details) might hinder or bias the recall of value-charged information

This type of memory, Value-Account:- is more accessible than episodic memorycan be used to cue the reconstruction

from episodic memory- is better retained over time

People may recall an overall evaluation of an event even when they fail to recall contextual details - “I like it but I don’t know why”

24

Constructive Value-Account Control (no-graphing)

Constructive iScale, but not the Value-Account, performed better than control condition

•More experience reports•With more details (references to temporal information, discrete events)•Higher test-retest consistency of time estimation (i.e., when did an experience take place)

•Higher test-retest consistency of graphed patterns (over Value-Account)

Technology Assisted Reconstruction

Newcastle, 8 March 2011

Trajectory reminders Emosnaps Footprint tracker

Experience Sampling Method

a few relevant venues would still miss a substantial number of studies. We instead used the ACM Guide to Computing Literature querying for the term “experience sampling” without constraining to particular venues. This query returned 284 papers, published in more than fifty venues.

Figure 2. 243 papers referring to experience sampling over a ten-year period. Retrieval took place on August 26th, 2011.

Fourty-one papers were excluded from further processing. These were: references to proceedings of conferences (n=13), papers that dealt with research problems that were beyond the scope of HCI such as computer networks, transport economics and organizational psychology (n=16), papers where no traces of the term “experience sampling” could be found (n=6), most likely caused by errors in the retrieval process such as due to stemming algorithms, papers that were written in languages other than English (n=3), papers that used the term ‘experience sampling’ to refer to concepts other than the method (n=2), and a single paper that was found to be a duplicate entry.

The remaining 243 papers were further classified with respect to being in one of the following types (Interrater agreement of two coders on a random subset of 60 papers, Fleisch K=0.74): empirical studies using the experience sampling method (n=59), empirical studies that considered but decided against the use of ESM (n=21), tools for carrying out experience sampling studies (n=20), systems that borrow concepts from ESM (n=10), papers discussing particular methodological aspects of ESM such as psychometric scales (n=13), reflection papers discussing conceptual issues of in-situ measurement (n=17), and papers discussing ESM as prior (n=85) or future (n=18) work.

For the scope of this paper we will analyze in detail only two categories of papers:

a) 59 papers reporting empirical studies that used the experience sampling method, and

b) 21 papers reporting empirical studies that considered, but decided against the use of the experience sampling method.

Out of the 59 papers that reported empirical studies, eight were found to report a study that was published earlier, and two cited incorrectly the use of experience sampling

0

15

30

45

60

2001 2003 2005 2007 2009 2011

method while following a user-initiated diary approach. The analysis of the remaining 49 studies is being reported below.

Study length, sampling frequency, and response rateThe majority (80%) of the studies had a duration of several days up to one month with 14 studies (34%) lasting between four and seven days (see figure 3). Only two studies had a duration of more than a month.

Figure 3. Distribution of 41 experience sampling studies over their length. Eight papers did not report study length.

Thirty nine papers reported the particular sampling approach that was followed in the study. Eighteen of these studies (46%) followed a random sampling approach. Often researchers would assign constraints with regard to the minimum and/or maximum (e.g. [64]) temporal distance between two samples, or the maximum samples taken during a day (e.g. [12]). In seven studies (18%) the sampling was scheduled for particular moments, while the remaining 14 (36%) followed an event-based sampling approach, triggered by incoming call handling in the user’s mobile phone [29], changes in the user’s location [25], attempts to connect to a wifi-access point [44] and other user interactions. The frequency of sampling varied considerably across studies, ranging from 10 minutes to a couple of hours, with most studies that spanned throughout the day to limit sampling frequency to less than 10 per day. Previous research has revealed that a sampling frequency of five to eight times per day may yield an optimal balance of recall and annoyance, while sampling at predetermined times is preferred over random as interruptions can be anticipated [45].

Only 11 papers reported users’ response rate to experience samples. Reported response rate was high, ranging from 17% to 93% (median=71%). Some papers reported varying response rate depending on the sampled day with weekdays displaying higher response rate (63%) than days of the weekend (55%) [52], or the type of questions with qualitative ones, requiring free-text responses, displaying lower response rate (73%) as opposed to quantitative such as scales and selection tasks (89%) [2].

0

5

10

15

≤1h < 24h ≤ 3d ≤ 1w ≤ 2w ≤ 1m > 1m

3

No of papers referring to Experience Sampling

What variables do ES studies measure?We distinguish below between self-reported measures of behavior and experience (see Table 1). This distinction is relevant as their reconstruction follows a different process whereas behavioral information may be directly accessible through episodic memory while experiential information has to be further inferred from recalled episodic cues [63].

Table 1. Number of ES studies eliciting self-reported measures of behavior, experience, or both.

Type of measures that studies elicit No

Self-reported measures of behavior 5

Self-reported measures of experience 22

Self-reported measures of behavior & experience 18

Behavioral measures related most frequently to the activity (n=18) that the participant was engaged with prior to being interrupted (e.g. [30]), its duration (n=2) (e.g. [35]), the participant’s current physical location (n=15) (e.g. [17]) and the social context (n=10), e.g. the number or nature of relationship of people that are in close proximity or participate with in a conversation (e.g. [34]). Other measures of behavior related to mode of transit [25], participants’ current physical engagement [18] and mode of convrersation (e.g. f2f, fixed/mobile phone etc.) [32].

Experiential measures related to:• Attitudes towards behaviors or events (n=15) such as

being interrupted (e.g. [55]), disclosing information to relevant others (e.g. [17]), or being video recorded (e.g. [58]).

• Measures of affect and experience (n=18) such as mental engagement (e.g. [18]) and concentration [14], satisfaction [27], mood and emotional states (e.g. [53], [23], experienced stress [61] and fun [14], interpersonal connectedness [19] and telepresence [14].

• Cognitive judgments (n=10) relating to stimuli, such as judging the relevance or usefulness of an advert [64], ranking the features of the product under use [2], making credibility assessment of provided information [62], forming judgments of the outcome of an undertaken activity [49], or one’s self-efficacy on the sampled activity [4].

• Motivations for exhibited behaviors (n=9) such as charging one’s mobile phone [6], handling incoming calls [28], posting information on social networking sites [50], tracking the location of family members [49].

What are the most frequent reasons for not selecting ESM?Twenty-one papers reported empirical studies that considered, but decided against the use of the experience

sampling method. Two of these provided no justifications for their choice. Analyzing the remaining 19 papers resulted to a total of 11 reasons for choosing alternative methods to ESM (see table 2).

Table 2. Reasons for not selecting the Experience Sampling Method along with frequency of occurence (No of papers).

Reason No

Disrupts the activity 6

Imposes high burden to participants 3

Requires high effort from researchers 3

Inappropriate for eliciting rich qualitative data 3

Misses rare and brief events 3

The user should be in control of when, what and how often to report

2

Limits sample size 2

Depends on participants’ ability to articulate ongoing experience

2

Poses privacy concerns 2

Limits number of measured variables 1

Technology limitations 1

As expected, the most frequent reason for not selecting the ES method was the interruptions that the method imposes on the user’s activity (eg. [48], [5]). For instance, Lindley and Monk [48] attempted to inquire into the social experiences of storytelling and reminiscing while revisiting photos with friends. While ESM would enable the momentary assessment of these experiences, it would have a strong impact on group behavior. Instead, they argued that behavioral measures can be used to infer aspects of these social experiences.

Other reasons for selecting alternative methods were the high burden that ESM imposes on participants, especially when typing in mobile situations is required (e.g. [47]), but also the high effort required to run the study (e.g. [31]). Three papers reported a need for eliciting rich qualitative accounts as a reason for choosing an alternative method such as Day Reconstruction diaries combined with self-reported experience narratives [41] and “lightweight capture of memory-trigger media in-the-moment, followed by in-depth annotation and review from a PC” [42].

4

Prompts at random, or computationally estimated times, to self-report on ongoing behaviors and experiences.

– Where are you?– What are you doing?– How far is your mobile phone? – How do you feel?

Day Reconstruction Method

produce the exact past event, but instead, every attempt to recall results in a new, often altered representation of the event. Bartlett asked participants to recall an unfamiliar story that they were told 20 hours before. Recalled stories differed from the original one in missing details, altering the order and importance of events, or in applying rationalizations and interpretations to the original story. Stories were further distorted through repeated reconstruction.The notion that remembering is an act of reconstruction instead of mere reproduction has received wide support. At the heart of reconstruction lies the distinction between episodic and semantic memory [69]. While episodic memory “is specific to a particular event from the past, semantic memory is not tied to any particular event but rather consists of certain generalizations (i.e. beliefs) that are rarely updated” (p. 935). These two types of memory serve different needs such as learning new information quickly - a capacity of episodic memory - or developing relatively stable expectations about the world - a capacity of semantic memory [63]. Reconstruction happens through the retrieval of cues from episodic memory. In the absence of contextual cues in episodic memory, beliefs found in semantic memory may be used to reconstruct the past, resulting in distortions such as the ones found in Barlett’s study. Thus, overall, the accuracy of one’s remembered events lies in the degree to which contextual cues are still present in the person’s episodic memory.

But, how do we reconstruct emotional experiences that contain not only contextual details of the experienced event, but also value-charged information such as emotions or overall evaluative judgments on the event?

Robinson and Clore [63] argue that “emotional experience can neither be stored nor retrieved” (p. 935), but can only be reconstructed on the basis of recalled contextual cues. They propose an accessibility model that distinguishes between four types of knowledge used to construct an emotion (see figure 1). First, experiential knowledge is used when an emotion is constructed online, i.e. as the experience takes place. It is considered the only “authentic” source of information, yet it is accessible for only a small fraction of time, about 3 seconds [37]. Self-reports tapping into experiential information, such as ESM, are considered the “gold standard” of experience measurement [36] as they avoid biases caused by retrospection.

When experiential knowledge is inaccessible, people will resort to episodic information, i.e. they will recall contextual cues from episodic memory and infer their emotions on the basis of these contextual cues. For instance, in recalling one’s emotions while experiencing a roller coaster, one might recall having his hands raised and screaming, and on this basis to infer an aroused experience. The accuracy with which people may recall their emotions will thus depend on the richness of recalled episodic

information. Daniel Kahneman and colleagues [36] relied on this proposition to develop the day reconstruction method. DRM asks participants to list their daily activities as a continuous list of episodes [36]. This is assumed to form stronger temporal and semantic links across the distinct experiences, thereby minimizing bias from overly relying on semantic information, which is detached from the actual experience. It has been shown that DRM provides a reasonably good approximation to experience sampling data, both in between-subjects [36] and within-subject [20] analyses. When episodic memories become inaccessible, people will shift to semantic memory. People will first access situation-specific beliefs, i.e. “a belief about the emotions that are likely to be elicited in a particular type of situation” [63]. If event-specific beliefs are inaccessible, e.g. due to rarity of the event, people will access identity-related beliefs, i.e. “beliefs about their emotions in general”. Such information is detached from the actual experience and is associated with retrospection biases found in global reports of experience such as online surveys and questionnaires used at the end of empirical studies [65].

A REVIEW OF EXPERIENCE SAMPLING AND DAY RECONSTRUCTION STUDIESThe experience sampling method has been popular in HCI for over a decade. Yet, to our knowledge, no attempt has been made to review the use of the method by HCI researchers. In the following section we attempt a review of empirical studies, published within HCI, that employ either the experience sampling or the day reconstruction method. The questions we will attempt to adress are: How are these methods used within HCI? What are the benefits these methods offer in addressing particular HCI problems? And, what drawbacks are experienced in practice?

Experience samplingAs ES studies have been published across a wide range of HCI venues we decided that narrowing our search down to

2

Figure 1. Four sources of information in emotional self-report according to Robinson and Clore [63]. Figure adapted from [63].

Experiential Knowledge

Online emotion, e.g.Experience Sampling

Episodic memory

Situation-specific belief

Identity-related belief

Retrospective, e.g.Day Reconstruction

Exit questionnaires

Type of Knowledge Source of information Type of Self-Report

Exit questionnaires

Semantic

Semantic

Episodic

Episodic

Can a retrospective method help participants in recalling more accurately their experiences?

Kahneman et al. (2004)

Robinson & Clore (2002)

Trajectory reminders in location-based preferences

Can self-face pictures assist in recalling momentary emotions?If so, is it through a recognition or a reconstruction process?

Footprint tracker

Technology Assisted ReconstrucHon

Emosnaps - inferring emotion from self-face pics


Do trajectory reminders (locations visited before and after) increase the test-retest reliability of the reconstruction process?

How do visual cues (i.e., Sensecam), location cues, and context cues (SMS and calls made or received) assist in reconstructing daily behaviors and experiences?



Footprint tracker






! !

When recalling with trajectory reminders, participants were more consistent over two repeated recalls separated by 1 week

Control condition With trajectory reminders



Footprint tracker






Self-Cam ’06Teeters, Kaliouby & Picard

Measuring emoHons in mobile contexts

Wearable EMG interface ’10Gruebler & Suzuki

Can we develop a tool that is truly unobtrusive to daily life and can be employed in long-term field studies?



Time-Daycontrol condition

Photo-dayEmotion reconstruction

Photo-weekEmotion recognition

Experience Sampling (Ground truth)

78% of pictures could be used for inferring emotions



Footprint tracker






1. Sensecam

2. Location logging3. Context logging

(SMS/calls made or received)

User experience over timeWhat makes for positive experiences in the long run?

iScale Can an online survey tool assist users in recollecting their experiences with a product?

Tech. Assist. ReconstructionCan mobile sensors assist participants in reconstructing their daily experiences and whereabouts?

Thank youEvangelos Karapanos

ekarapanos.com