Usability Testing Kathryn Summers ©2010

Usability TestingUsability Testing

Kathryn Summers©2010

Copyright 2003 Kathryn Summers

Usability TestingPreparation

RecruitingScreeningTasks and Scenarios

TestingFacilitationObservation

AnalysisPresentation

Benefits to User-Centered Products (2/09)

Users find them more useful (can do more tasks that the user cares about)

Users can do things more efficientlyUsers learn how to use them more easily

and use more oftenUsers like them more (less frustrating,

more enjoyable

Costs of Poor Design


Costs to clientincreased support costsreduced salesexpensive redesign

Costs to customer/userreduced employee productivityincreased employee frustrationmore frequent mistakes

Some general usability principles


Simple and natural language (user’s language and labels)

visual consistency (graphics, navigation, layout, labeling)

appropriate feedbackerror prevention, error recoveryhelp information

And information design principles


Information available right when and where it’s needed

Organization and presentation of information reflects user’s needs and goals

Visual design reinforces task flows and logical information groupings

Types of tests: Evaluative Testing


Confirm that you’ve met a benchmark for usability before release

Confirm that product is more usable than a prior version or a competing product

Evaluative testing will focus on tasks and may often involve quantitative measurements

Types of tests: Exploratory testsTesting prototypes, or comparing prototypes


Focus on exploring relationship between system image and user’s mental modelRepresenting classes of objectsRepresenting relationships between objectsAllowing user to manipulate objects

Test navigation, help access, subject matter organization

Values of UsabilityReal world/real user perspectiveTends to find serious functional issues

Values of Eye-trackingAllows testers to see if users read or not –

amount of time spent in area of screen vs scanning

Has value in decisions on layout/ navigation/labeling

Effective user testing


Testing the right peopleTesting the right tasks

User testing vs. market research vs. “testing the software”

Watching with attention


What users expect from a designHow they respond

Most understandable ways to present content: How the website should be structuredExact function,

arrangement of function, and system

Does the website actually work for the people who are supposedly going to use it? Why or why not?

Determine who is coming online in your targetDemographicPsychographicAttitudes and preferences

Determine what information will provide value to your targetDistinct information needs

and preferences by target sub-segment

Information gaps and priorities

Tools and resources

Market ResearchUser Research

Preparation: Recruiting Participants

Identify your target user groups (demographics, background knowledge, experience, relationship to site/tool, goals)

Decide which groups you want, and how many users you’ll recruit

(decide what characteristics users will share, what characteristics will vary)

Draft a test participant screener (what are make/break characteristics)

Test the screener

Recruiting Participants Decide on number of participants Over recruit (dropout/no show rate) Determine compensation

(amount, cash vs. gift, depends on time spent, organization type, project type)

Acquire consent

Preparation: Developing the Test Identify test objectives/create list of problems Translate objective/problem list into tasks Prioritize tasks (frequency, importance), decide

which tasks to test, Develop scenarios for tasks Write test script (scenarios) Identify resources participants will need for the

tasksHardware, software, data files, instructions, internet connection, time

Determine what data/measurements to collect (metrics)

Conduct a pilot test

Tasks vs. Scenarios Tasks are series of actions made to

achieve goal – ie., “request article from Interlibrary Loan using online form.”

Scenarios are tasks put into a short narrative – ie., “your instructor recommends an article and gives you a citation for “Usability Tests made Simple,” from The Journal of Usability, 2 (34), 11-23. You realize that Cook Library doesn’t have that journal, and you would like to request it from Interlibrary Loan”.

They are meant to take some of the artificiality out of the task.

Tasks vs. ScenariosScenarios should: Allow user to SHOW not TELL actions Include one and only one task Have defined set of pathways to complete

task Use natural language (user’s, not

system’s) Be short – avoid excessive background,

irrelevant details Be clear and provide enough instruction

detail to complete the task (test scenarios with pilot users)

Tasks vs. ScenariosScenarios should NOT: Walk user through process

(e.g. You have the afore-mentioned article citation. Go to the Interlibrary Loan link located in the list of services on the left. Complete the online form including author, article title, journal title, volume, issue and page number)

Scenario should test user’s ability to complete the task in a natural way without step-by-step instructions.

Preparation: Writing scriptWhy a script?

Ensures consistencyScript should include:

Explanation of testConsent (if not done beforehand)Think aloud protocolTasks (description if written, scenarios if

verbal)Questionnaire/demographic questionsFollow up if appropriate

Preparation: Establishing rolesFacilitator

ScriptObserver(s)

Actions Metrics CommentInferences

Observers can capture metrics in metrics-based testing


time on taskerror ratesuccessful completion ratenumber of times (or how long) help system

is accessed time spent recovering from errorsnumber of commands/features used

Preparation: Pilot testTest script to see if scenarios/tasks are

workingTest equipmentPractice roles for moderator and observers

Preparation: MaterialsParticipant ListConsent formsCompensation/acknowledgementsScriptsTask scenarios (written/part of script)Observation forms

Facilitating the test Explain the test Help the test participant feel comfortable

(emphasis on testing SYSTEM not USER) Model/Encourage a think-aloud protocol Listen and observe – avoid “rescuing” Probe (avoid using leading questions) Respect user’s expertise

Becoming a better moderator


Become a “people” personBe calm and flexiblePay close attentionWatch details but within larger context

Cohesive picture of each testPatterns between tests

Don’t just rely on memoryCommunicate results effectivelyStay organizedPractice!!!

Watching with attention


Try not to test your own forms/designsTry to watch with a completely open mindWatch carefully—pay attention to body

language, facial expressions, sounds, where the user is looking, and what they do

Pay most attention to behavior, then to explanations, least attention to preferences or predictions about what other users want

Most of all…


Understand that the user never makes “mistakes”—the user only makes efforts to solve the problem

Your goal is to understand what problem-solving effort the user is making

Analysis during the test: Keep a problem list


Keep a free-hand list during the test—note “critical incidents” that you may ask follow-up questions about at the end of the session

Use the “highlighter” methodHave a printout of interface screens on hand; circle problem areas as you observe them during the tests

Watch the tapes, categorizing related problems after the test

Capture metrics if desired

ReferencesGoldberg, J. H., Stimson, M. J., Lewenstein, M., Scott, N., Wichansky, A. M. (2002).

Eye tracking in web search tasks: Design implications. In Proceedings of the 2002 symposium on Eye tracking research & applications (pp. 51-58). Retrieved March 28, 2009 from ACM Digital Library.

Hackos, J. T. & Redish, J. C. (1998). User and task analysis for interface design. New York: John Wiley & Sons.

Jeffries, R., Miller, J. R., Wharton, C., & Uyeda, K. M. (1991). User interface evaluation in the real world: Comparison of four techniques. In Proceedings of the SIGCHI conference on human factors in computing systems: Reaching through technology (pp. 119-124). Retrieved March 28, 2009 from ACM Digital Library.

Molich, R., Ede, M. R., Kaasgaard, K., Karyukin, B., (2004). Comparative usability evaluation. Behavior & Information technology, 23 (1), 65. 74. Retrieved March 28 from Academic Search Premier database.

Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests. New York: John Wiley & Sons.

Snyder, C. (2003). Paper prototyping: The fast and easy way to design and refine user interfaces. San Francisco: Morgan Kaufman.

Limitations of usability testing


Testing situations are always artificial (maybe in a lab, assigned tasks, paid participation)

Test participants are rarely perfectly representative users

The test design can never fully duplicate natural user behaviorResults depend heavily on tasks—we see

what we think to look for

Finding participants with lower literacy skills


Try to recruit participants that Have a strong motivation to care about the

interface (e.g., users with a particular health condition)

Read at 8th grade level or below (REALM 60 or below)

Have been observationally screened for a minimum level of computer ability Open browserNavigate to website using address bar OR

searchScrollUse linksUse mouse

Where to recruit users with lower literacy skills


Recruit fromHealth clinicsLiteracy centersPublic libraries

Plan for a two-step recruiting approachAdminister the REALM; observe computer useSchedule selected users for testing

REALM (Rapid Estimate of Adult Literacy in Medicine) (Davis, et al., Family Medicine 1993 25.6: 391-5.)

Documents

Usability Testing Kathryn Summers ©2010