1
Evaluation / Usability
Usability Is it a “good” interface?
In what ways? Usability:
How well users can use the system’s functionality
How well can the interface be used – it’s just that simple
5Original
6Proximity
7Repetition
Examples of Bad Design … and Why
Elevator controls and labels on the bottom row all look the same, so it is easy to push a label by mistake instead of a control button
People do not make same mistake for the labels and buttons on the top row. Why not?
Logical or ambiguous design?
Where do you plug the mouse?
Where do you plug the keyboard?
top or bottom connector?
Do the color coded icons help?
From: www.baddesigns.com
Internal and external consistency Internal consistency refers to designing
operations to behave the same within an application Difficult to achieve with complex interfaces
External consistency refers to designing operations, interfaces, etc., to be the same across applications and devices Very rarely the case, based on different designer’s
preference Most successful in product families (e.g MS Office) Op. Sys. vendors may define style guidelines
External Inconsistency …
Keypad numbers layout
1 2 34 5 6
7 8 9
7 8 9
1 2 3
4 5 6
0 0
(a) phones, remote controls (b) calculators, computer keypads
Usability Problem Example: Unexpected Occurrence of Events
Usability Measures – 5 Often Used Time to learn
How long does it take for typical members of the community to learn relevant task?
Speed of performance How long does it take to perform relevant benchmarks?
Rate of errors by users How many & what kinds of errors are made during
benchmark tasks? Retention over time
Frequency of use and ease of learning help make for better user retention
Subjective satisfaction Do they like it? Allow for user feedback via interviews, free-form
comments and satisfaction scales
16Original
compliant interaction
state evident inmechanical buttons
rotary knobs reveal internal state and can be controlled by both user and
machine
Evaluation TechniquesEvaluation
tests usability and functionality of system occurs in laboratory, field and/or in
collaboration with users evaluates both design and
implementation
19
Questionnaire Design Collect general demographic information that
may be relevant Age, sex, computer experience, etc.
Evaluate feelings towards your product and other products
Important to design questionnaire carefully Users may find questions confusing
May not answer the question you think you are asking May not measure what you are interested in
20
Questionnaire, 2 “Likert scale”
Propose something and let people agree or disagree:agree disagree
The system was easy to use: 1 .. 2 .. 3 .. 4 .. 5 “Semantic differential scale”
Two opposite feelings:difficult easy
Finding the right information was: -2 .. -1 .. 0 .. 1 .. 2 If multiple choices, rank order them:
Rank the choices in order of preference (with 1 being most preferred and 4 being least): Interface #1 Interface #2 Interface #3 Interface #4 (in a real survey, describe the interfaces)
21
Survey example
Subject ID: ____________
An Experimental Study on various input techniques
Questionnaire
Please complete the following questionnaire. Please feel free to ask any
questions.
1. Please rate each input method. 1. Mouse
terrible very poor poor OK good very good great
-3 -2 -1 0 1 2 3 2. Tapping on the SmartBoard
terrible very poor poor OK good very good great
-3 -2 -1 0 1 2 3 3. Using a Laser Pointer in one hand with the button in the other
terrible very poor poor OK good very good great
-3 -2 -1 0 1 2 3 4. Using a Laser Pointer built into a Palm (Symbol) device
terrible very poor poor OK good very good great
-3 -2 -1 0 1 2 3 5. Using a stylus on the PocketPC device
terrible very poor poor OK good very good great
-3 -2 -1 0 1 2 3
6. Remote control joystick provided with the projector
1. Please rank order your preference, using 1 for most preferred and 6 for least: Mouse Tapping on the SmartBoard Using a Laser Pointer in one hand with the button in the other Using a Laser Pointer built into a Palm (Symbol) device Using a stylus on the PocketPC device Remote control joystick provided with the projector
2. Please rate your proficiency in using computers.
Never used before Novice Expert
0 1 2 3 4 5 6 7
3. Please rate your proficiency in using a Palm Pilot or other PDA.
Never used before Novice Expert
0 1 2 3 4 5 6 7
4. Please rate your proficiency in using a laser pointer in general (not necessarily tracked by the computer).
Never used before Novice Expert
0 1 2 3 4 5 6 7
5. Please rate your proficiency in using a touch sensitive whiteboard like the SmartBoard.
Never used before Novice Expert
0 1 2 3 4 5 6 7
6. Do you own a Palm Pilot or other Personal Digital Assistant (PDA)? If so, which model?
Yes, I have a:
No
7. Please supply some information about yourself:
Male Female Age: 8. Please write down any other comments or suggestions for improvement: __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________ __________________________________________________________________________
- End -
22
Where Evaluate? Usability Labs
Cameras, 2-way mirrors,specialists
Separate observationand control room Should disclose who is watching
Having one may increase usability evaluations in an organization
Can usually perform an evaluation anywhere Can use portable videotape recorder, etc.
Cognitive Walkthrough
Proposed by Polson et al. evaluates design on how well it supports
user in learning task usually performed by expert in cognitive
psychology expert ‘walks though’ design to identify
potential problems using psychological principles
Cognitive Walkthrough
Example: Code walkthrough in Software Engineering
To review a segment of program code by the expert other than programmer
Sequence of actions Selecting a set of program codes Check certain characteristics, such as, coding style,
spelling variables conventions, function declarations, system-wide invariants etc.
Cognitive Walkthrough (ctd)
For each task walkthrough considers what impact will interaction have on user? what cognitive processes are required? what learning problems may occur?
Analysis focuses on goals and knowledge: does the design lead the user to generate the correct goals?
26
Questions Cognitive Walkthrough
1. Is the next goal clear at this stage?2. Is the appropriate action obvious?3. Is it clear that this action leads to the goal?4. What problems are there in performing the
action?
Cognitive Walkthrough: How (cont.) Walk through the task while answering these
Questions: Will the user know what to do? Will the user see how to do it? Will the user understand from feedback whether
their action was correct?
Heuristic Evaluation Proposed by Nielsen and Molich.
usability criteria (heuristics) are identified design examined by experts to see if these are
violated Example heuristics
system behaviour is predictable system behaviour is consistent feedback is provided
Heuristic evaluation `debugs' design.
The Procedure Several independent evaluators
each uses the same checklist each works alone each makes a list of usability problems
Combine lists into a single list works well as a group activity
HE: 10 Usability Heuristics
1. Visibility of system status The system should always keep users informed about what is going
on, through appropriate feedback within reasonable time
2. Match between system and the real world The system should speak the users' language, with words, phrases and
concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order
3. User control and freedom Users often choose system functions by mistake and will need a
clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo
HE: 10 Usability Heuristics4. Consistency and standards
Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions
5. Error prevention Even better than good error messages is a careful design which
prevents a problem from occurring in the first place
6. Recognition rather than recall Make objects, actions, and options visible. The user should not have
to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate
HE: 10 Usability Heuristics7. Flexibility and efficiency of use
Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions
8. Aesthetic and minimalist design Dialogues should not contain information which is irrelevant or rarely
needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility
9. Help users recognize, diagnose, and recover from errors Error messages should be expressed in plain language (no codes),
precisely indicate the problem, and constructively suggest a solution
HE: 10 Usability Heuristics10. Help and documentation
Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large
Jacob Nielsen originally developed the heuristics for heuristic evaluation in collaboration with Rolf Molich in 1990. Nielsen since refined the heuristics based on a factor analysis of 249 usability problems to derive a set of heuristics with maximum explanatory power, resulting in this revised set of heuristics
34
“Think Aloud” Protocols “Single most valuable usability engineering method” Find out why user does things
What thought would happen, why stuck, frustrated, etc. Encourage users to expand on whatever interesting But interferes with timings May need to “coach” user to keep talking
Unnatural to describe what thinking Ask general questions: “What did you expect”, “What
are you thinking now” Not: “What do you think that button is for”, “Why didn’t you click
here” Will “give away” the answer or bias the user
Alternative: have two users and encourage discussion
Physiological measurements emotional response linked to physical
changes these may help determine a user’s reaction
to an interface measurements include:
heart activity, including blood pressure, volume and pulse.
electrical activity in brain: electroencephalogram (EEG)
Eye tracking head or desk mounted equipment tracks the
position of the eye eye movement reflects the amount of cognitive
processing a display requires measurements include
fixations: eye maintains stable position. Number and duration indicate level of difficulty with display
saccades: rapid eye movement from one point of interest to another
scan paths: moving straight to a target with a short fixation at the target is optimal
Reality …
39
Example:prompts
What is a DTIC user code and how to get one?
40
Example: prompts (Print)
Keystroke-Level Model (KLM)
KLM OperatorsK Press a key or button
P Point to a target on the display
H Home hands on input device
D Draw a line segment
M Mentally prepare for an action
R (system response time)
Example: Replacing a Word
Operator EstimatesKeystroke determined by typing speed0.28s for average typist (40 wpm), 0.08s for fast typist (155 wpm), 1.20s for worst typist
Pointing determined by Fitts’ Law (or general approximation)T = a + b log (d/s +1) ORT = 1.1s
Drawing determined by Steering LawT = a + b (d/s)
Operator EstimatesHoming estimated by measurementT = 0.36s (between keyboard and mouse)
Mental prep estimated by measurementT = 1.35s(estimated by taking the total task time, subtracting physical operator time, and dividing by the number of “chunks” of activity)
Adapted from Rob Miller
Example: Deleting a WordUsing Shift-Click
MP [start of
word]K [click]MP [end of word]K [shift]K [click]H [to keyboard]MK [Del]
Total: 3M + 2P + 4K= 7.37 sec
Using DeleteMP [start of word] K [click]HMK [Del] x n [length of word]
Total: 2M + P + H + (n+1) K= 4.44 + 0.28n sec
In-class ExerciseGenerate a KLM model for deleting a file from your desktop
Estimate the time it would take using the provided operator times
Compare the predicted time with the actual time
48
Example
Consider the searching a Word document for all occurrences of a four-letter word, and replacing it with another four-letter word.
49
Reach for mouse H[mouse] 0.40Move pointer to "Replace" button P[menu item] 1.10
Click on "Replace" command K[mouse] 0.20
Home on keyboard H[keyboard] 0.40Specify word to be replaced M4K[word] 2.15
Reach for mouse H[mouse] 0.40Point to correct field P[field] 1.10Click on field K[mouse] 0.20Home on keyboard H[keyboard] 0.40Type new word M4K[word] 2.15Reach for mouse H[mouse] 0.40Move pointer on Replace-all P[replace-all] 1.10
Click on field K[mouse] 0.20Total 10.2
50
Error Messages Blame the system, not the user
“Unrecognized” vs. “illegal” command Easy error recovery Can have multiple levels of messages E.g. in XXX product, “can't save file” — why
not?
Problem: Geek-speak: Refers to “Print dialog”, a technical UI component term
Microsoft Word 98Print Status Dialog Box
Microsoft Word 98Print Status Dialog Box
Problem: Two buttons (stop/cancel) to end the print??
Eudora Pro 4.0 Special menu
Problem: What’s the difference between Find and Search?
Virus Scan 3.0.3 Warning
Problem: Careless writing. Note typo -- “more that”
Microsoft Notepad Menu
Problem: No keyboard equivalents
Microsoft Windows Ports Control Panel
Problem: Dialog box control buttons mixed with setting and help buttons
CS5540
Eudora Version Installation Dialog Box
Problem: Traps user: no way to Cancel installation at this point. Gives options for bad modes.
CS5540 58
SpellTools User Dictionary
…Problem: No “...” on Import button
CS5540 59
Oracle’s ORS CLEATS web-application
Problem: Should write out “Pin Number,” not “Pin No”
Unsubscribe from mailing list
Problem: Silly message
61
Bad Error Messages
62
More bad error messages!
63
Another Bad Example
http://stinet.dtic.mil/
64
Another Bad Example
65
Another Bad Example
66
Pretty Good Example Pretty Good: travel.yahoo.com: Says what to do to fix it
But language is inconsistent
67
Help and Documentation True walk up and use? Most people will not read documentation
If do, then First time product is used, or else In a panic, so need information right away
Iterative design of documentation needed SuperBook application answer found in 4.3 minutes,
compared to 7.6 minutes before fixing Help system is an extra feature to learn “Help doesn’t”
If need to add help, maybe fix the feature? Use documentation writers to help refine the system
Good quality writing
68
Good Help Example
NSFreportsystem What
& Why
69
Firefox: No Zoom Indicator
70
Worst interfaces
71
72
Kitchen timer
73
Turn down the TV
74
Xerox Real Business (worst 2011)
Examples of non-usable systems
EclipseHard learning curve, bad for beginnersToo many windows, confusing layout, too much going on (advanced options)
ISIS pops up a window, pop-up blocker problems can’t use browser’s back button gives course instructor number instead of name must enter schedule-number instead of clicking on course ID and
number Must use semester as a code, and if you get this wrong it’s
confusing what’s wrong Interface has pictures/images for links, not the conventional
under-lined links not integrated into COD, so must copy info over from COD to
ISIS Kicks you off too quickly Too often overloaded, too many users Unavailability at certain hours / don’t know when it’s unavailable /
status not visible easily Links on main page are too similar SSN number use
77
175 design Guidelines These are primarily intended to guide you to a good design, not to be used for
evaluation1) Help users plan goals, tasks2) Provide a clear model of how users view system in terms of tasks3) Help users with system model, metaphors, work context4) Design to match user’s conception of high-level task organization5) Help users understand what system features exist and how they can be used in their
work context6) Help users decompose tasks logically7) Make clear all possibilities for what users can do at every point8) Keep users aware of system state for planning next task9) Keep the task context visible to minimize memory load10) Help users plan the most efficient ways to complete their tasks11) Keep users aware of task progress, what’s been done and what’s left to do12) Provide cognitive affordances at the end of critical tasks to remind users to
complete the transaction13) Provide effective cognitive affordances that help users get access to system
functionality14) Help users know/learn what actions are needed to carry out intentions
Quoted from Chapter 19 of H. Rex Hartson and Pardha S. Pyla, The UX Book: Ensuring a Quality User Experience, to be published by Morgan Kaufmann / Elsevier in 2011.
Guidelines, cont.15) Help users know how to do something at action/object level16) Help users predict outcome of actions17) Help users determine what to do to get started18) Provide a cognitive affordance for a step the user might forget19) Support user with effective sensory affordances in presentation of cognitive
affordances20) Make cognitive affordances visible21) Make cognitive affordances noticeable22) Make text legible, readable23) Control cognitive affordance presentation complexity with effective layout,
organization, and grouping24) Present cognitive affordance in time for it to help the user before the
associated action25) Help user determine actions with effective content/meaning in cognitive
affordances26) Design cognitive affordances for clarity27) Use precise wording in labels, menu titles, menu choices, icons, data fields28) Use a verb and noun and even an adjective in labels where appropriate.29) Avoid vague, ambiguous terms. 78
Quoted from Chapter 19 of H. Rex Hartson and Pardha S. Pyla, The UX Book: Ensuring a Quality User Experience, to be published by Morgan Kaufmann / Elsevier in 2011.
Guidelines, cont.30) Be as specific to the interaction situation as possible31) Clearly represent work domain concepts32) Use dynamically changing labels when toggling33) Provide cognitive affordances to indicate formatting within data fields34) Constrain the formats of data values to avoid data entry errors35)Provide clearly marked exits36)Provide clear “do it” mechanism37)Be predictable; help users predict outcome of actions with feed-forward
information in cognitive affordances38)Make choices distinguishable39)Be consistent with cognitive affordances40)Use consistent wording in labels for menus, buttons, icons, fields41)Use similar names for similar kinds of things42)Do not use multiple synonyms for the same thing43)Use the same term in a reference to an object as the name or label of the
object44)Use different terms for different things, especially when the difference is
subtle 79Quoted from Chapter 19 of H. Rex Hartson and Pardha S. Pyla, The UX Book: Ensuring a Quality User Experience, to be published by Morgan Kaufmann / Elsevier in 2011.
Guidelines, cont.
80
45) Be consistent in the way that similar choices or parameter settings are made
46) Decompose complex instructions into simpler parts47) Use appropriate layout and grouping by function to convey content and
meaning48) Group together objects and design elements associated with related tasks
and functions49) Do not group together objects and design elements that are not associated
with related tasks and functions50) Support user choices with likely and useful defaults51) Provide the most likely or most useful default selections52) Offer most useful default cursor position53) Relieve human short term memory loads by maintaining task context
visibly or audibly for the user54) Support human memory limits with recognition over recall55) Avoid requirement to retype or copy from one place to another56) Support special human memory needs in audio interaction design57) Avoid cognitive indirectness58) Be complete in your design of cognitive affordances; include enough
information for users to determine correct actionQuoted from Chapter 19 of H. Rex Hartson and Pardha S. Pyla, The UX Book: Ensuring a Quality User Experience, to be published by Morgan Kaufmann / Elsevier in 2011.
Guidelines, cont.
81
59) Prevent loss of productivity due to hesitation, pondering60) Use enough words for unambiguous labels61) Add supplementary information, if necessary62) Give enough information for users to make confident decisions63) Give enough alternatives for user needs64) Employ usage-centered wording, the language of the user and the work
context, in cognitive affordances65) Find ways to anticipate and avoid user errors in your design66) Help users avoid inappropriate and erroneous choices67) Disable buttons, menu choices to make inappropriate choices unavailable68) Gray out to make inappropriate choices appear unavailable69) But help users understand why a choice is unavailable70) Provide a clear way to undo and reverse actions71) Offer constructive help for error recovery72) Avoid confusing modalities73) Distinguish modes clearly74) Use “good modes” where they help natural interaction without confusion75) Support human memory limitations in the design of task structure76) Support user with effective task structure and interaction control77) Provide alternative ways to perform tasks
Quoted from Chapter 19 of H. Rex Hartson and Pardha S. Pyla, The UX Book: Ensuring a Quality User Experience, to be published by Morgan Kaufmann / Elsevier in 2011.
Guidelines, cont.
82
78) Provide shortcuts79) Provide logical grouping in layout of objects80) Group together objects and functions related by task or user work activity81) But avoid grouping of objects and functions if they need to be dealt with
separately82) Support task thread continuity by anticipating the most likely next task,
step, or action83) Make the most of user’s work84) Do not requiring users to re-enter data85) Retain user state information86) Avoid the feeling of loss of control87) Give direct manipulation support88) Always provide a way for the user to “bail out” of an on-going operation89) Support users making physical actions with effective sensory affordances
for sensing physical affordances90) Support user with effective physical affordances for manipulating objects,
help in doing actions91) Avoid physical awkwardness92) Accommodate physical disabilities93) Design layout to support manual dexterity and Fitts’ law
Quoted from Chapter 19 of H. Rex Hartson and Pardha S. Pyla, The UX Book: Ensuring a Quality User Experience, to be published by Morgan Kaufmann / Elsevier in 2011.
Guidelines, cont.
83
94) Support targeted cursor movement by making selectable objects large enough
95) Group clickable objects related by task flow close together96) But not too close, and do not include unrelated objects in the grouping97) Design physical movement to avoid physical overshoot98) Include physicality in your design when the alternatives are not as
satisfying to the user99) Check your functionality for missing features100)Check your functionality for non-user-interface software bugs101)Avoid too much automation and loss of user control102)Help the user by automating where there is an obvious need103)Provide feedback for all user actions104)Provide progress feedback on long operations105)Request confirmation as a kind of intervening feedback106)But don’t overuse and annoy107)Support user with effective sensory affordances in presentation of
feedback108)Make feedback visible109)Make feedback noticeable110)Locate feedback within the user’s focus of attention
Quoted from Chapter 19 of H. Rex Hartson and Pardha S. Pyla, The UX Book: Ensuring a Quality User Experience, to be published by Morgan Kaufmann / Elsevier in 2011.
Guidelines, cont.
84
111) Make feedback large enough to notice112)Make text legible, readable113)Control feedback presentation complexity with effective layout,
organization, and grouping114)Help users detect error situations early115)Maintain a consistent appearance across similar kinds of feedback116)Maintain a consistent location of feedback presentation on the screen to
help users notice it quickly.117)Use the most effective feedback presentation medium118)Consider audio as alternative channel119)Help users understand outcomes with effective content/meaning in
feedback120)Design feedback for clarity121)Support clear understanding of outcome (system state change), so users
can assess effect of actions122)Give clear indication of error conditions123)Be complete in your design of feedback; include enough information for
users to fully understand outcomes and be either confident that their command worked or certain about why it didn’t
124)Prevent loss of productivity due to hesitation, ponderingQuoted from Chapter 19 of H. Rex Hartson and Pardha S. Pyla, The UX Book: Ensuring a Quality User Experience, to be published by Morgan Kaufmann / Elsevier in 2011.
Guidelines, cont.
85
125)Add supplementary information, if necessary126)Give enough information for users to make confident decisions about the
status of their course of interaction127)Help users understand what the real error is128)Give enough information about the possibilities or alternatives so user can
make an informed response to a confirmation request129)Design feedback wording, especially error messages, for positive
psychological impact130)Make the system take blame for errors131)Be positive, to encourage132)Provide helpful, informative error messages, not “cute” unhelpful messages133)Employ usage-centered wording, the language of the user and the work
context, in displays, messages, and other feedback134)Be consistent with feedback135)Label outcome or destination screen or object consistently with starting
point and action136)Organize feedback for ease of understanding137)Provide user control over amount and detail of feedback138)Give only most important information at first; more on demand
Quoted from Chapter 19 of H. Rex Hartson and Pardha S. Pyla, The UX Book: Ensuring a Quality User Experience, to be published by Morgan Kaufmann / Elsevier in 2011.
Guidelines, cont.
86
139)Organize information displays for ease of understanding140)Eliminate unnecessary words141)Group related information142)Control density of displays; use white space to set off143)Columns are easier to read than wide rows144)Use abstraction per Shneiderman’s “mantra”: Overview first; zoom and
filter; details on demand145)Employ usage-centered wording, the language of the user and the work
context146)Avoid the use of anthropomorphism in interaction designs147)Avoid using first-person speech in dialogue148)Avoid condescending offers to help149)Avoid poor attempts at humor150)Avoid violent, negative, demeaning terms151)Avoid use of psychologically threatening terms, such as “illegal”, “invalid”,
“abort”152)Avoid use of the term “hit” instead of “press” or “click”153)Avoid irritation with annoying sound and color in displays154)Use color conservatively155)Use pastels, not bright colors
Quoted from Chapter 19 of H. Rex Hartson and Pardha S. Pyla, The UX Book: Ensuring a Quality User Experience, to be published by Morgan Kaufmann / Elsevier in 2011.
Guidelines, cont.
87
156)Be aware of color conventions (e.g., avoid red, except for urgency)157)Watch out for focusing problem with red and blue158)Avoid fancy or cute design without a real purpose159)Make presentation of text legible160)Make font size large enough for all users161)Use good contrast with background162)Use mixed case for extensive text163)Avoid too many different fonts, sizes164)Use legible fonts165)Use color other than blue for text166)Accommodate sensory disabilities and limitations167)Allow user settings, preference options to control presentational
parameters168)Accommodate different levels of expertise/experience with preferences169)Don’t let affordances for new users be performance barriers to experienced
users170)Be helpful with Help171)Do not try to achieve the appearance of simplicity by just reducing
usefulness172)Organize complex systems to make the most frequent operations simple
Quoted from Chapter 19 of H. Rex Hartson and Pardha S. Pyla, The UX Book: Ensuring a Quality User Experience, to be published by Morgan Kaufmann / Elsevier in 2011.
Guidelines, cont.
88
173)Use consistent layout/location for objects across screens174)Maintain custom style guides to support consistency175)Use structurally similar names and labels for objects and functions that are
structurally similar176) ….and the list isn’t even finished
Quoted from Chapter 19 of H. Rex Hartson and Pardha S. Pyla, The UX Book: Ensuring a Quality User Experience, to be published by Morgan Kaufmann / Elsevier in 2011.