IN4MATX 231 IN4MATX 231 Human-Computer Human-Computer

InteractionInteraction

Presenters:Presenters:

Ritesh Ritesh SubramanianSubramanian

Tanmay GoelTanmay Goel

Observing UsersObserving Users

OutlineOutline

Benefits and challenges of different Benefits and challenges of different types of observations.types of observations.

How to observe as an onlooker, a How to observe as an onlooker, a participant, and an ethnographer.participant, and an ethnographer.

How to collect, analyze and present How to collect, analyze and present observational data.observational data.

Indirect observation using diary Indirect observation using diary studies and logging.studies and logging.

Qualitative and Quantitative analysis.Qualitative and Quantitative analysis.

What and when to observeWhat and when to observe

Observation is usable at all stages during Observation is usable at all stages during product development.product development.

Goals and questions determine the Goals and questions determine the paradigms and techniques used.paradigms and techniques used.

Observers can be:Observers can be:– OnlookersOnlookers– ParticipantsParticipants– EthnographersEthnographers

The degree of immersion that evaluators The degree of immersion that evaluators adopt varies across a broad outsider-adopt varies across a broad outsider-insider spectrum.insider spectrum.

Types of ObservationTypes of Observation

ObservationObservationControlled Controlled

Environment Environment (e.g. Lab)(e.g. Lab)

Field Field Environment Environment (e.g. Natural)(e.g. Natural)

Outsider looking on

““Quick and Dirty”Quick and Dirty”

In usability In usability testingtesting

““Quick and Dirty”Quick and Dirty”

In field testingIn field testing

InsiderInsider N.AN.A..

Participant Participant observation (e.g. observation (e.g. in ethnography)in ethnography)

Approaches to ObservationApproaches to Observation

““Quick and Dirty” observationQuick and Dirty” observation– It can occur anywhere, anytimeIt can occur anywhere, anytime– Least formality involvedLeast formality involved

Observation in usability testingObservation in usability testing– Video and interaction logs capture all the Video and interaction logs capture all the

user operationsuser operations– One-way mirrors or remote TV screenOne-way mirrors or remote TV screen– Data is used to analyze what users are Data is used to analyze what users are

doing and provide insight into users’ doing and provide insight into users’ affective reactionsaffective reactions

Approaches to ObservationApproaches to Observation

Observation in field studiesObservation in field studies– The observer may be a passive observer, participant The observer may be a passive observer, participant

observer or an ethnographer.observer or an ethnographer.– Goal is to cause as little disruption as possible.Goal is to cause as little disruption as possible.– Passive or outside observer – An observer in a class of Passive or outside observer – An observer in a class of

boys and girls whose primary job to keep track of how boys and girls whose primary job to keep track of how much time a particular technology is used by each much time a particular technology is used by each gender.gender.

– Participant observer – Participates in social conventions Participant observer – Participates in social conventions of a group, combines participation with maintenance of of a group, combines participation with maintenance of professional distance for unbiased observation.professional distance for unbiased observation.

– Ethnography – Takes weeks, months or longer. Inside Ethnography – Takes weeks, months or longer. Inside information of community activity is obtained.information of community activity is obtained.

How to observe:How to observe:In controlled environmentIn controlled environment

Practical issues that need to thought of in advance:Practical issues that need to thought of in advance:– Decide where users are located so that equipment Decide where users are located so that equipment

can be setup in advancecan be setup in advance– Decide about modes of data capture e.g. video, Decide about modes of data capture e.g. video,

interaction logsinteraction logs– Equipment testing to get expected performance Equipment testing to get expected performance

(audio and video)(audio and video)– Get legal user consentGet legal user consent– Prepare a script to guide users through the set of Prepare a script to guide users through the set of

questions questions – It is important to make users feel comfortableIt is important to make users feel comfortable– Problem with this approach: Observers do not know Problem with this approach: Observers do not know

what users are thinking.what users are thinking.

Controlled Environment – Controlled Environment – Think-Aloud TechniqueThink-Aloud Technique

The user is trying to perform a certain The user is trying to perform a certain task. Observer wonders, what is going task. Observer wonders, what is going on, what is he thinking, why did he do on, what is he thinking, why did he do that, etc.that, etc.

Eternalize thought process.Eternalize thought process. A big problem is occurrences of silence A big problem is occurrences of silence

during think-aloud process.during think-aloud process. Maybe have two people work together Maybe have two people work together

and talk to each other.and talk to each other.

Controlled Environment –Controlled Environment –Think Aloud TechniqueThink Aloud Technique

Checklist of things to plan Checklist of things to plan before going into the fieldbefore going into the field

State the initial goal and questions clearlyState the initial goal and questions clearly Select a framework to guide activities in the fieldSelect a framework to guide activities in the field Decide how to record eventsDecide how to record events Think about how to gain acceptance and trust of users Think about how to gain acceptance and trust of users

under observationunder observation Be prepared to go through these notes to weed out Be prepared to go through these notes to weed out

ambiguitiesambiguities Highlight and separate personal opinion from what Highlight and separate personal opinion from what

happenshappens Be prepared to refine and refocus the study for same or Be prepared to refine and refocus the study for same or

new user groupsnew user groups Think about how to handle sensitive issuesThink about how to handle sensitive issues Consider working as a team; conform notes with another Consider working as a team; conform notes with another

membermember Consider checking notes with an informant and plan to Consider checking notes with an informant and plan to

look at the situation from a different perspectivelook at the situation from a different perspective

How to observe (contd.):How to observe (contd.):In the fieldIn the field

Frameworks to guide observationFrameworks to guide observation– The personThe person. Who? . Who? – The place.The place. Where? Where?– The thing.The thing. What? What?

The Goetz and LeCompte (1984) framework:The Goetz and LeCompte (1984) framework:– WhoWho is present? is present? – WhatWhat is happening? is happening? – WhenWhen does the activity occur? does the activity occur?– WhereWhere is it happening? is it happening? – WhyWhy is it happening? is it happening? – HowHow is the activity organized? is the activity organized?

In the fieldIn the field

The Robinson (1993) frameworkThe Robinson (1993) framework– SpaceSpace. What is the physical space like?. What is the physical space like?

– ActorsActors. Who is involved?. Who is involved?

– ActivitiesActivities. What are they doing?. What are they doing?

– ObjectsObjects. What objects are present? . What objects are present?

– ActsActs. What are individuals doing?. What are individuals doing?

– EventsEvents. What kind of event is it?. What kind of event is it?

– GoalsGoals. What do they to accomplish?. What do they to accomplish?

– FeelingsFeelings. What is the mood of the group and of . What is the mood of the group and of individuals?individuals?

How to observe (contd.):How to observe (contd.):Participant observer or ethnographyParticipant observer or ethnography

Checklist for doing ethnography Checklist for doing ethnography – Identify the problem or goal and ask good Identify the problem or goal and ask good

questions to be answered by the studyquestions to be answered by the study– The most important part of fieldwork is being there The most important part of fieldwork is being there

to observe, ask questions and record what is seen to observe, ask questions and record what is seen and heardand heard

– Collect variety of data. E.g. Notes, still pictures, Collect variety of data. E.g. Notes, still pictures, audio and videoaudio and video

– Be prepared to move backwards and forwards Be prepared to move backwards and forwards between the broad picture and specific questions between the broad picture and specific questions

– Analyze the data using a holistic approach in which Analyze the data using a holistic approach in which observations are contextualizedobservations are contextualized

Data Collection TechniquesData Collection Techniques

Notes plus still cameraNotes plus still camera– The least technical way of collecting dataThe least technical way of collecting data– Transcription the first step in data analysisTranscription the first step in data analysis– Photographs, sketches, etc. can be easily collectedPhotographs, sketches, etc. can be easily collected

Audio recording plus still cameraAudio recording plus still camera– Less intrusive than video Less intrusive than video – More flexible and mobile More flexible and mobile – One drawback is transcribing the data which can One drawback is transcribing the data which can

be onerous if many hours have to be transcribedbe onerous if many hours have to be transcribed

Data Collection TechniquesData Collection Techniques

VideoVideo– Advantage of both audio and Advantage of both audio and

video data; but more intrusivevideo data; but more intrusive– Attention becomes focused on Attention becomes focused on

what is seen through the lens what is seen through the lens – Analysis of video can be long Analysis of video can be long

and time consumingand time consuming– Sound may get muffled when Sound may get muffled when

recording in noisy conditionsrecording in noisy conditions

ComparisonComparison

Table source: Preece, Rogers, Sharp: Interaction Design: Beyond Human-Computer Interaction. Wiley

Comparison (contd.)Comparison (contd.)

Table source: Preece, Rogers, Sharp: Interaction Design: Beyond Human-Computer Interaction. Wiley

Indirect observation: Indirect observation: tracking user activitiestracking user activities

DiariesDiaries– Diaries provide a record of what users did, when Diaries provide a record of what users did, when

they did it and what they thought about their they did it and what they thought about their interactions with technologyinteractions with technology

– Useful when users and scattered and unreachableUseful when users and scattered and unreachable– Inexpensive and require no special equipment or Inexpensive and require no special equipment or

expertiseexpertise– They rely on participants being reliable and They rely on participants being reliable and

remembering to complete themremembering to complete them– Participants usually remember incidents for better Participants usually remember incidents for better

or for worseor for worse

Indirect observation: Indirect observation: tracking user activitiestracking user activities

Interaction loggingInteraction logging– Includes analysing data from key presses, mouse Includes analysing data from key presses, mouse

movements etc.movements etc.– Usually synchronized with video and audio logsUsually synchronized with video and audio logs– Time stamped to calculate how much time a user Time stamped to calculate how much time a user

spends on a particular taskspends on a particular task– E.g. Explicit counters that record visits to a E.g. Explicit counters that record visits to a

website. If you want to find out if adding a bulletin website. If you want to find out if adding a bulletin board increases the number of visits, comparison board increases the number of visits, comparison of traffic before and after is usefulof traffic before and after is useful

– Unobtrusive and large volumes of data can be Unobtrusive and large volumes of data can be collectedcollected

Interaction Logging- Interaction Logging- Webtrends Log Analyzer Webtrends Log Analyzer

Interaction LoggingInteraction Logging

Ethical Concerns: They don't know Ethical Concerns: They don't know we are watching. Shall we tell them?we are watching. Shall we tell them?

If we tell users about logging they may react or If we tell users about logging they may react or change their behaviorchange their behavior

It depends on the context, and how much personal It depends on the context, and how much personal information is collectedinformation is collected

e.g. Companies tell about data logged for quality e.g. Companies tell about data logged for quality assurance purposesassurance purposes

Concerns arise when personal information is used Concerns arise when personal information is used to infer financial or health informationto infer financial or health information

Analyzing, interpreting and Analyzing, interpreting and presenting the datapresenting the data

Three main types of data are exploredThree main types of data are explored QualitativeQualitative datadata - - interpretedinterpreted & used to tell the & used to tell the

‘story’ about what was observed‘story’ about what was observed. . Qualitative dataQualitative data - - categorizedcategorized using techniques using techniques

such as content analysis.such as content analysis. Quantitative dataQuantitative data - collected from interaction & - collected from interaction &

video logs. Presented as values, tables, charts, video logs. Presented as values, tables, charts, graphs and treated statistically.graphs and treated statistically.

Qualitative analysis to tell a Qualitative analysis to tell a storystory

Review the data after each observation to Review the data after each observation to identify key themes identify key themes

Record themes in a coherent yet flexible form Record themes in a coherent yet flexible form – E.g. Laptops, audio recordingsE.g. Laptops, audio recordings

Record the date and time of each data sessionRecord the date and time of each data session As themes emerge, check your understanding As themes emerge, check your understanding

with informants or people you observewith informants or people you observe Iterate until the story reflects observationIterate until the story reflects observation Report findings to a group in the form of a Report findings to a group in the form of a

presentation or written reportpresentation or written report

Analyzing and reporting Analyzing and reporting ethnographic dataethnographic data

Look for key events within a group that drive a Look for key events within a group that drive a group’s activitygroup’s activity

Look for patterns or behavior among different Look for patterns or behavior among different playersplayers

Compare sources of data against each other Compare sources of data against each other Report findings in a convincing and honest wayReport findings in a convincing and honest way e.g. Software tools such as Ethnograph allow e.g. Software tools such as Ethnograph allow

ethnographers to code their notes so that they can be ethnographers to code their notes so that they can be searched, sorted and retrieved. The information can searched, sorted and retrieved. The information can be printed out as a tree showing relationships of be printed out as a tree showing relationships of occurrences. occurrences.

EthnographEthnograph

EthnographEthnograph

Qualitative analysis for Qualitative analysis for categorizationcategorization

Looking for incidents or patternsLooking for incidents or patterns– Look for critical incidents, such as times when the Look for critical incidents, such as times when the

user was obviously stuck. (marked by silence, looks user was obviously stuck. (marked by silence, looks of confusion)of confusion)

– E.g. Jurgen Koenemann-Belliveau et al(1994) used E.g. Jurgen Koenemann-Belliveau et al(1994) used this approach to compare the efficacy of two this approach to compare the efficacy of two versions of a Smalltalk programming manual for versions of a Smalltalk programming manual for novice programmers.novice programmers.

– They were able to identify specific problems by They were able to identify specific problems by tracing through sequence of incidents and thereby tracing through sequence of incidents and thereby achieving a holistic understanding.achieving a holistic understanding.

Qualitative analysis for Qualitative analysis for categorizationcategorization

Analyzing data into categoriesAnalyzing data into categories– Content analysis provides a reliable and systematic way Content analysis provides a reliable and systematic way

of coding content into meaningful sets of mutually of coding content into meaningful sets of mutually exclusive categoriesexclusive categories

– Categories should be orthogonal i.e. no overlapCategories should be orthogonal i.e. no overlap– Appropriate granularity must be selected Appropriate granularity must be selected – e.g. Training two people to use categories and letting e.g. Training two people to use categories and letting

them analyze the same data sample. If huge discrepancy them analyze the same data sample. If huge discrepancy in the analyses is observed, then either the training was in the analyses is observed, then either the training was insufficient or categories need to be redefinedinsufficient or categories need to be redefined

– Inter-research reliability rating- Percentage of agreement Inter-research reliability rating- Percentage of agreement between two researchers, defined as ratio of number of between two researchers, defined as ratio of number of items categorized in the same way to the total number of items categorized in the same way to the total number of items observeditems observed

Qualitative analysis for Qualitative analysis for categorizationcategorization

Analyzing DiscourseAnalyzing Discourse– Focus on the dialog, i.e. meaning of what was said Focus on the dialog, i.e. meaning of what was said

rather than contentrather than content– Discourse analysis is strongly interpretiveDiscourse analysis is strongly interpretive– Language is viewed as a constructive tool and Language is viewed as a constructive tool and

discourse analysis provides a way of focusing upon discourse analysis provides a way of focusing upon people’s use of language to construct versions of people’s use of language to construct versions of their worldstheir worlds

– E.g. Analyzing discourse on the Internet (e.g. in E.g. Analyzing discourse on the Internet (e.g. in chat rooms, bulletin boards) has started to chat rooms, bulletin boards) has started to influence designers’ understanding about user influence designers’ understanding about user needs in these environmentsneeds in these environments

Quantitative data analysis Quantitative data analysis and feedbackand feedback

Quantitative data analysisQuantitative data analysis– Video data collected is usually annotated to calculate Video data collected is usually annotated to calculate

performance times performance times – The data stream from interaction logs is used to calculate The data stream from interaction logs is used to calculate

performance timesperformance times– Data is analyzed using means, standard deviations, t-Data is analyzed using means, standard deviations, t-

tests etc.tests etc. Feeding the findings back into the designFeeding the findings back into the design

– Written reports make for easy reading and a good Written reports make for easy reading and a good reference documentreference document

– Include anecdotes, pictures, quotations and video clips to Include anecdotes, pictures, quotations and video clips to bring study to life and stimulate interestbring study to life and stimulate interest

– Verbal presentations can be a powerful mode of feedbackVerbal presentations can be a powerful mode of feedback

SummarySummary

Observation in usability study tends to be objectiveObservation in usability study tends to be objective In contrast, in participant observation the evaluator works In contrast, in participant observation the evaluator works

with the user to understand their activities and beliefswith the user to understand their activities and beliefs Ethnography uses a technique wherein the ethnographer Ethnography uses a technique wherein the ethnographer

immerse themselves in the culture they studyimmerse themselves in the culture they study Observational data collection depends on the paradigm Observational data collection depends on the paradigm

usedused Combination of video, audio, diaries and logs can be used Combination of video, audio, diaries and logs can be used

to collect observational datato collect observational data Analyzing video and data logs can be tedious due to the Analyzing video and data logs can be tedious due to the

sheer volume of datasheer volume of data Evaluators flag events in real time and return to examine Evaluators flag events in real time and return to examine

them later. them later. Identifying key events is an effective approachIdentifying key events is an effective approach