AVI/Psych 358/IE 340: Human Factors

EvaluationOctober 31, 2008

2

Why, what, where and when to evaluate

Iterative design & evaluation is a continuous process that examines:– Why: to check that users can use the product and

that they like it. – What: a conceptual model, early prototypes of a

new system and later, more complete prototypes.– Where: in natural and laboratory settings.– When: throughout design; finished products can be

evaluated to collect information to inform new products.

3

Evaluation approaches

• Usability testing

• Field studies

• Analytical evaluation (e.g., think aloud)

4

Characteristics of approaches

Usability testing

Field studies

Analytical

Users do task natural not involved

Location controlled natural anywhere

When prototype early prototype

Data quantitative qualitative problems

Feed back measures & errors

descriptions problems

Type applied naturalistic expert

5

Evaluation approaches and methods

Method Usability testing

Field studies

Analytical

Observing x x

Asking users

x x

Asking experts

x x

Testing x

Modeling x

6

Usability testing of a website

• MedlinePlus: Provides health information for public, doctors, and other medical professionals

• Created by National Library of Medicine (NLM)

7

Goals and questions

• To identify the range of usability problems– Navigation– Information seeking– Etc.

8

Selection of participants

• Nine subjects from primary health care practices through recruitment posters in Washington, DC

• Subjects completed brief questionnaire (age, web experience, frequency of seeking health-related information)

• 7 were women, 2 were men• Gender balance?

9

Development of tasks

1. Find information about whether a dark bump on your shoulder might be skin cancer

2. Find information about whether it is safe to use Prozac during pregnancy

3. Find information about whether there is a vaccine for hepatitis C

4. Find recommendations about the treatment of breast cancer, specifically the use of mastectomies

5. Find information about the dangers associated with drinking alcohol during pregnancy

10

Procedure (1/2)

• Five scripts were prepared for the user test– Introduction to the study– Consent form– List of tasks– Post-experiment questionnaire

11

Procedure (2/2)

• Same instructions were given to every subject• Testing done in laboratory-like conditions

– Subjects sat in front of a monitor alone– Were allowed to explore website for 10 minutes prior to tasks– Think aloud protocol used, 20 minutes for each of the five

tasks– If task was not finished, they were asked to stop (ensuring

equal time for each task)

12

Data collection

• Video• Actions on website were logged• Start and completion time• Page counts• Medical publications accessed• Negative comments• User satisfaction questionnaire

13

Data analysis

• Website organization

• Browsing efficiency

• Use of search features

• …

14

Results

• Reaching external sites was often difficult• Users experienced difficult finding health topics

devoted to different types of cancer• High ease of learning but low ease of use

15

Usability testing (1/2)

• Involves recording performance of typical users doing typical tasks

• Controlled environmental settings

• Users are observed and timed

16

Usability testing (2/2)

• The data is used to calculate performance times, and to identify & explain errors

• User satisfaction is evaluated using questionnaires & interviews

• Field observations may be used to provide contextual understanding

17

Running Experiments

• Experiments test hypotheses to discover new knowledge by investigating the relationship between two or more things – i.e., variables.

18

Usability lab with observers watching a user & assistant

19

Portable equipment for use in the field

20

21

Testing conditions

• Usability lab or other controlled space.• Emphasis on:

– selecting representative users;– developing representative tasks.

• 5-10 users typically selected.• Tasks usually last no more than 30 minutes.• The test conditions should be the same for every

participant.• Informed consent form explains procedures and deals

with ethical issues.

22

Some type of data Time to complete a task. Time to complete a task after a specified. time away

from the product. Number and type of errors per task. Number of errors per unit of time. Number of navigations to online help or manuals. Number of users making a particular error. Number of users completing task successfully.

23

How many participants are enough for user testing?

• The number is a practical issue.• Depends on:

– schedule for testing;– availability of participants;– cost of running tests.

• Typically 5-10 participants. • Some experts argue that testing should continue until no

new insights are gained.