Gathering Usability Data

Observing users & subjective data

Directing Sessions

Issues:Are you in same room or not?Single person session or pairs of peopleObjective data -- stay detached

Fall 2006 PSYCH / CS 6750 2

Collecting Data

Data gatheringNote-takingAudio and video tapeInstrumented user interfacePost-experiment questions and interviews

Fall 2006 PSYCH / CS 6750 3

Collecting Data Identifying errors can be difficultQualitative techniques

Think-aloud - can be very helpfulPost-hoc verbal protocol - review videoCritical incident logging - positive & negativeStructured interviews - good questions

“What did you like best/least?”“How would you change..?”

Fall 2006 PSYCH / CS 6750 4

Observing Users

Not as easy as you think

One of the best ways to gather feedback about your interface

Watch, listen and learn as a person interacts with your system

Fall 2006 PSYCH / CS 6750 5

Observation

Direct In same room Can be intrusive Users aware of your

presence Only see it one time May use 1-way mirror to

reduce intrusiveness

Indirect Video recording Reduces intrusiveness,

but doesn’t eliminate it Cameras focused on

screen, face & keyboard Gives archival record,

but can spend a lot of time reviewing it

Fall 2006 PSYCH / CS 6750 6

Location

Observations may beIn lab - Maybe a specially built usability lab

Easier to controlCan have user complete set of tasks

In fieldWatch their everyday actionsMore realisticHarder to control other factors

Fall 2006 PSYCH / CS 6750 7

Challenge

In simple observation, you observe actions but don’t know what’s going on in their head

Often utilize some form of verbal protocol where users describe their thoughts

Fall 2006 PSYCH / CS 6750 8

Verbal Protocol

One technique: Think-aloudUser describes verbally what s/he is thinking and

doingWhat they believe is happeningWhy they take an actionWhat they are trying to do

Fall 2006 PSYCH / CS 6750 9

Think Aloud

Very widely used, useful techniqueAllows you to understand user’s thought

processes better

Potential problems:Can be awkward for participantThinking aloud can modify way user performs task

Fall 2006 PSYCH / CS 6750 10

Teams

Another technique: Co-discovery learning (Constructive interation)Join pairs of participants to work togetherUse think aloudPerhaps have one person be semi-expert (coach)

and one be noviceMore natural (like conversation) so removes some

awkwardness of individual think aloud

Fall 2006 PSYCH / CS 6750 11

Alternative

What if thinking aloud during session will be too disruptive?

Can use post-event protocolUser performs session, then watches video

afterwards and describes what s/he was thinkingSometimes difficult to recallOpens up door of interpretation

Fall 2006 PSYCH / CS 6750 12

Historical Record

In observing users, how do you capture events in the session for later analysis??

Fall 2006 PSYCH / CS 6750 13

Capturing a Session

1. Paper & pencilCan be slowMay miss thingsIs definitely cheap and easy

Fall 2006 PSYCH / CS 6750 14

Time 10:00 10:03 10:08 10:22

Task 1 Task 2 Task 3 …

Se

Se

Capturing a Session

2. Recording (audio and/or video)Good for talk-aloudHard to tie to interfaceMultiple cameras probably neededGood, rich record of sessionCan be intrusiveCan be painful to transcribe and analyze

Fall 2006 PSYCH / CS 6750 15

Capturing a Session

3. Software loggingModify software to log user actionsCan give time-stamped key press or mouse eventTwo problems:

Too low-level, want higher level eventsMassive amount of data, need analysis tools

Fall 2006 PSYCH / CS 6750 16

IssuesWhat if user gets stuck on a task?You can ask

“What are you trying to do..?”“What made you think..?”“How would you like to perform..?”“What would make this easier to accomplish..?”Maybe offer hints

Can provide design ideas

Fall 2006 PSYCH / CS 6750 17

Subjective Data

Satisfaction is an important factor in performance over time

Learning what people prefer is valuable data to gather

Fall 2006 PSYCH / CS 6750 18

Methods

Ways of gathering subjective dataQuestionnairesInterviewsBooths (eg, trade show)Call-in product hot-lineField support workers

(Focus on first two)

Fall 2006 PSYCH / CS 6750 19

Questionnaires

Preparation is expensive, but administration is cheap

Oral vs. writtenOral advs: Can ask follow-up questionsOral disadvs: Costly, time-consuming

Forms can provide better quantitative data

Fall 2006 PSYCH / CS 6750 20

Questionnaires

IssuesOnly as good as questions you askEstablish purpose of questionnaireDon’t ask things that you will not useWho is your audience?How do you deliver and collect questionnaire?

Fall 2006 PSYCH / CS 6750 21

Questionnaire Topic

Can gather demographic data and data about the interface being studied

Demographic data:Age, genderTask expertiseMotivationFrequency of useEducation/literacy

Fall 2006 PSYCH / CS 6750 22

Interface Data

Can gather data aboutscreengraphic designterminologycapabilitieslearningoverall impression...

Fall 2006 PSYCH / CS 6750 23

Question Format

Closed formatAnswer restricted to a set of choicesTypically very quantifiableVariety of styles

Fall 2006 PSYCH / CS 6750 24

Closed Format

Likert ScaleTypical scale uses 5, 7 or 9 choicesAbove that is hard to discernDoing an odd number gives the neutral choice in

the middle

Fall 2006 PSYCH / CS 6750 25

Characters on screen

hard to read easy to read 1 2 3 4 5 6 7

Other Styles

Fall 2006 PSYCH / CS 6750 26

LaTeX

FrameMaker

WordPerfect

Word

Rank from1 - Very helpful2 - Ambivalent3 - Not helpful0 - Unused

___ Tutorial___ On-line help___ Documentation

Which word processingsystems do you use?

Closed Format

Advantages Clarify alternatives Easily quantifiable Eliminate useless answer

Disadvantages Must cover whole range All should be equally

likely Don’t get interesting,

“different” reactions

Fall 2006 PSYCH / CS 6750 27

Open Format

Asks for unprompted opinionsGood for general, subjective information, but

difficult to analyze rigorouslyMay help with design ideas

“Can you suggest improvements to this interface?”

Fall 2006 PSYCH / CS 6750 28

Questionnaire Issues

Question specificity“Do you have a computer?”

LanguageBeware of terminology, jargon

ClarityLeading questions

Can be phrased either positive or negative

Fall 2006 PSYCH / CS 6750 29

Questionnaire Issues

• Prestige bias – People answer a certain way because they want

you to think that way about them

• Embarrassing questions• Hypothetical questions• “Halo effect”

– When estimate of one feature affects estimate of another (eg, intelligence/looks)

Fall 2006 PSYCH / CS 6750 30

Deployment

StepsDiscuss questions among teamAdminister verbally/written to a few people

(pilot). Verbally query about thoughts on questions

Administer final test

Fall 2006 PSYCH / CS 6750 31

Interviews

Get user’s viewpoint directly, but certainly a subjective view

Advantages:Can vary level of detail as issue arisesGood for more exploratory type questions which

may lead to helpful, constructive suggestions

Fall 2006 PSYCH / CS 6750 32

Interviews

DisadvantagesSubjective viewInterviewer can bias the interviewUser may not appropriately characterize usageTime-consuming

Fall 2006 PSYCH / CS 6750 33

Interview Process

How toPlan a set of questions (provides for some

consistency)Don’t ask leading questions

“Did you think the use of an icon there was really good?”

Can be done in groupsGet consensus, get lively discussion going

Fall 2006 PSYCH / CS 6750 34

Data Analysis

Simple analysisDetermine the means (time, # of errors, etc.) and

compare with goal values (coming up…)

DetermineWhy did the problems occur?What were their causes?

Fall 2006 PSYCH / CS 6750 35

Experimental Results

How does one know if an experiment’s results mean anything or confirm any beliefs?

Example: 20 people participated, 11 preferred interface A, 9 preferred interface B

What do you conclude?

Fall 2006 PSYCH / CS 6750 36

Hypothesis Testing

In experiment, we set up a “null hypothesis” to checkBasically, it says that what occurred was simply

because of chance

For example, any participant has an equal chance of preferring interface A over interface B

Fall 2006 PSYCH / CS 6750 37

Hypothesis Testing

If probability result happened by chance is low, then your results are said to be “significant”

Statistical measures of significance levels0.05 often usedLess than 5% possibility it occurred by chance

Fall 2006 PSYCH / CS 6750 38

Presentation Techniques

Fall 2006 PSYCH / CS 6750 39

0 20

Mean

low highMiddle 50%

Time in secs.

Age

Upcoming

AudioWeb

Fall 2006 PSYCH / CS 6750 40

Using the Results

How do you use the results of your evaluation?

How can you make your design better with this knowledge?

Fall 2006 PSYCH / CS 6750 41