Upload
trancong
View
229
Download
6
Embed Size (px)
Citation preview
Evaluating and
Improving
Software Usability 902 : Thursday, 9:30am - 10:45am
Philip Lew
www.xbosoft.com
Understand, Evaluate and
Improve
2
Agenda
• Introduc7on
– Importance of usability
– What is usability?
– What is UX?
• Usability Modeling and Measurements
• Case studies on measuring and improving
• Summary
What is Quality?
Pertaining to soEware J
4
Requirements…
Importance of Usability
• Usability is important especially for web applica7ons
– And now especially mobile applica7ons
• Saas, mobile are in alignment with and soon becoming equivalent to cloud • Most prevalent implementa7on method in the next 5 years
• Without good usability:
– Users will leave the applica7ons
– For mobile, if they can’t learn in 30 seconds, they won’t come back
Why is Usability Important?
Web and Mobile have Changed the
Terrain • Business models
have changed
– Instead of paying
upfront and
‘owning’ the
soEware
– Pay as you go, pay
by subscrip7on
• Behavior and
expecta7ons have
changed
What is Usability
In rela7on to quality
In rela7on to user experience
When you think Usability…
• Naviga7on
• Efficiency
• Responsiveness-‐performance
• Learnability
• AZrac7veness
• Understandability
Anything
else come to
mind?
Current Research – Usability
Quality
Usability Satisfaction
ISO 9126-1
ISO 9241-11
Usability Design
Experts ISO 25010
User
Experience
Product Owners
Usability -‐ Key Characteris7c of
Product Quality
Source: ISO 25010
Quality in Use ISO 25010
Notes on Sa7sfac7on and Usability
• Sa7sfac7on is a subjec7ve feeling dependent on many things other than usability:
– A user can be highly sa7sfied but the applica7on with low usability.
– An applica7on can be highly usable (high usability) but the user is not sa7sfied!
Highly usable software
My password doesn’t work
don’t have what I want I’m unsatisfied
I’m satisfied!
! low usability
software
Finished my work today
Nice weather today
Usability-‐Major Component of Quality Quality and Usability in the so1ware development lifecycle
• Can be measured from the design point of
view or of the product
• Can be measured ‘in-‐use’ with real users
Usability
Usability
Evolu7on of SoEware quality
Software processes
Code
Process quality
Software Quality
(internal)
Software Quality
(external)
Software Quality
(in use)
Product ?
CMMI assessment
model
white box testing
black box testing ?
CMMI Assessment
Company
Programmer Tester End User
Type of quality
What is measured
How measured?
Who measures?
ISO 9000 ISO 9241 ISO 9126 ISO 25010
Source: ISO 25010
What is Usability-‐Effect of the
SoEware Product
Degree to which specified
users can achieve specified
goals with effectiveness,
efficiency and satisfaction in a
specified context of use.
Usability in Actual Usage
• User role
• Objec7ve
• Task
• Environment
• Domain
• …
specified users
specified goals
specified context of use
What else can you think of?
Usability
UX, Usability and Quality in Use • Difficult to understand their relationships
• Lack of consensus in meaning in literature
• Not totally clear how related to quality
• Standard definition for user experience is still not available
and User Experience
Some UX Defini7ons
UX DESCRIPTION
D1 a continuous process of user engagement with the product [10] D2 entire set of affects that results in user-product interaction [11] D3 the evolution of usability [12] D4 elaboration of the satisfaction component of usability [13] D5 a categorization of “do-goals” (pragmatics) and “be-goals”(hedonics) [3], [14]
D6 infinite small experiences relating to people, products and contexts [6] D7 consequence of user’s and product’s characteristics when interacted in a
specific environment [1] D8 degree to which specified users can achieve actual usability, safety, and
satisfaction in use in a specified context of use [15] D9 A person's perceptions and responses that result from the use or anticipated use
of a product, system or service [16]
D10 the degree of positive or negative emotions that can be experienced by a
specific user in a specific context during and after product use and that
motivates for further usage [17]
User Experience (UX)
• Pragmatic (do-goals) : refers to the product's
perceived ability to support the task achievement
and focuses on the product’s utility and usability in
completing tasks, the ‘do-goals’ of the user.
• Hedonic (be-goals) : product's ability to support
the user’s achievement of ‘be-goals’, such as
being satisfied, pleasure.
• Real goal of the end user is fulfilling be-goals such
as being autonomous, competent, related to
others, stimulated, etc.”
• Be-goals are the driver of UX
Hassenzahl
21
Rela7ng Usability and User Experience
Quality
characteris7c
508
compliancy Func7onality
User Interface
Usability Learnability
Product
aZributes
Text/Sound
aZributes
Appropriate
Func7ons Easy to use Easy to learn
UX Pragma7c
Do Goals Effec7ve and Efficient
UX Hedonic
Be Goals Sa7sfac7on (pleasure, sense of community…)
Usability
Measurement Effec7veness in use Efficiency in use Learnability in use
Sa7sfac7on
Measurement Pleasure Comfort
Sense of
Community Trust
Context
USER EXPERIENCE
Now that we understand it ALL…
We Need to Define Usability For Our
Own Purposes
22
Let’s Define Usability/UX
From the Product Viewpoint
Usability
Save Loca7on
Characteris7c 1 Characteris7c 2 Characteris7c n
Subcharacteris7c 1 Subcharacteris7c 2 Subcharacteris7c n
AZribute 1 AZribute 2 AZribute n
Understandable Naviga7on
Control Stability Previous-‐Next
Home Loca7on Ease of finding
Defining Usability from an
Effect-‐Real usage Point of View
Quality
Errors
Effec7veness Efficiency Characteris7c n
Accuracy Completeness Subcharacteris7c n
AZribute 2 AZribute 3 AZribute 1
Usability
Sa7sfac7on
Some Poten7al AZributes
Measurements for ‘in use’ Usability • Effec9veness
– Comple7on rates
– Error rate
– Help usage
• Efficiency
– Task 7me
– Backtracking
• Learnability
– Learning rate
– Task 7me devia7on
Can you think
of any others,
par9cular to
your
applica9on?
26
Adding in UX
Sa9sfac9on in use
• Degree to which users are sa7sfied in a specified context of use. Sa7sfac7on is the user’s response to interac7on with the product, including amtudes towards use of the product. Subdivided into sub-‐characteris7cs:
– Likability (cogni7ve sa7sfac7on)
– Pleasure (emo7onal sa7sfac7on)
– Comfort (physical sa7sfac7on)
– Trust
UX Component
External Quality Requirements (for
Shopping Cart Entity)
1 Usability
1.1 Understandability
1.1.1 Icon/label ease to be recognized
1.1.2 Information grouping cohesiveness
1.2 Learnability
1.2.1 ………………………………………………………..
1.3 Ease of Use
1.3.1 Control permanence
1.3.2 Helpfulness
Example of Product Quality Model
with AZributes (Operability in ISO 25010)
28
Learnability
• Degree to which the soEware product enables users to learn its applica7on -‐ 7 principles (ISO 9241-‐10)
– Suitability for the task -‐ should be suitable for the user’s task and skill level
– Self-‐descrip7veness-‐ should be clear what the user should do next
– Controllability -‐ the user should be able to control the pace and sequence of the interac7on
– Conformity with user expecta7ons -‐ should be consistent
– Error tolerant -‐ forgiving
– Suitability for individualiza7on -‐ should be able to be customized to suit the user
– Suitability for learning -‐ should support learning
29
Ease of use
• Degree to which the soEware product makes it easy for users to operate and control it.
– controllability
– error tolerance (by operator)
– conformity with user expecta7ons
30
Ease of Use -‐ Helpfulness
• Degree to which the soEware product
provides help when users need
assistance including help that is:
– easy to find
– comprehensive
– effec7ve
Example: Usability Measurement
31
ACribute
Scale How Calcula9on Goal Current
Help
completeness Percent of
Menu items
with
help
Compare
menus and
help items
% 90% 40%
Ease of
access Keystrokes
to
find/use a
feature/
func7on/
informa7on
Sample 50
items average 3.5 12
Consistency Number
loca7ons for
same buZon
Examine
menus and
doc.
integer 1 5
What types of usability measures are these?
32
Using a measurement model
Measurements
Metrics
Evalua7on
Measurement
Func7on
Indicator
Evalua7on Evalua7on
Sub-‐characteris7cs
Evalua7on Evalua7on Characteris7c
Usability
composed of
composed of
Usability-‐Review
• Can be measured from the design point of
view or of the product
• Can be measured ‘in-‐use’ with real users
usability usability in use
UX
34
Measuring Usability
Methods and models
Now that we know what it is, and
what to measure, then HOW?
Measurable AZributes
• Usability characteris7c
• Descrip7on and purpose
• How to measure
• What is measured
• Measurement/Calcula7on
• Range (min, max)
• Metric
• Objec7ve (goal) versus Current (indicator)
Once you have a
model (what you
are going to
measure), then
you start doing IT!
UX/Usability
Measurement Methods
Let’s get started
Logging
Heuris9c Evalua9on
Observa9on
Labs
Focus groups
Ques9onnaires
37
Measurement methods
Expert Evalua9on (Heuris9c evalua9on)
• Define a set of rules or criteria with measurements and evaluate against them
Web-‐based logs
• Collect user ac:vity data
– Mistakes and errors
– How long it takes to complete tasks
– Comple:on rates
Ques9onnaires
• Quan7ta7ve subjec7ve measurement of UX characteris7cs
Heuris7c Evalua7on
Using a checklist
But a liZle more complicated than
yes/no
38
39
Heuristic Evaluation of a Pharmacy
application Each usability attribute has a quantifying metric.
• For example, for Predic9ve textual aCribute, users should be able to understand a buCon’s results prior to pressing it.
• Direct metrics need to be designed, i.e.:
– (0) No support at all
– (1) Par9al
– (2) Complete
• Need a mapping from 0, 1, 2 to something more understandable, i.e. 2 = 100, and 1 = 60
• Need an indicator to interpret the level of sa9sfac9on met with decision criteria with acceptability ranges in a percentage scale:
– 0-‐40 (unsa9sfactory –red) means changes must take place with high priority;
– 40-‐70 (marginal –yellow) indicates a need for improvement ac9ons;
– 70-‐100 indicates a sa9sfactory level –green-‐ for the analyzed aCribute.
40
2.2.2 Error Recovery Support
" In the current state, users filling a new prescription are supported well in error recovery (2.2.2) in automatic cursor positioning
41
Heuristic Evaluation – Pharmacy Software The purpose of the evaluation • Understand the external quality level of the learnability sub-characteristic for
filling a new prescription
Table 2 - excerpt of the whole current evaluation
External Quality Requirements Measure EI value P/GI value
Global Quality Indicator 61.97%
1 Usability 60.88%
1.1 Understandability 83%
1.1.1 Icon/label ease to be recognized 100%
1.1.2 Information grouping cohesiveness 66%
1.2 Learnability 51.97%
1.2.1 ……………………………………………… …
1.3 Operability 49.50%
1.3.1 Control permanence 100%
1.3.2 Expected behaviour 50%
2 Content Quality 63.05%
2.1 Content Suitability 63.05%
2.1.1 Basic Information Coverage 50%
2.1.1.1 Line item information completeness 2 50%
2.1.1.2 Product description appropriateness 50%
2.1.2 Coverage of other Contextual Information 76.89%
2.1.2.1 ……………………………………………….. …
2.1.2.2 Return policy information completeness 33%
Heuris7c Evalua7on of Usability-‐Shopping Cart
JIRA Heuristic Usability Evaluation
Usability Logging
Collec7ng user ac7vity and behavior
Tradi7onally used for other things
like…
44
Usability Logging
• Iden7fy users by using session ID to iden7fy a unique user.
• Itera7vely insert code into the applica7on
• Collect data
• Analyze the data for each aZribute in different dimensions and aggrega7ons
• Determine the need for further calcula7ons and what aZributes to measure further
• Revise the data we are collec7ng, adding or decreasing granularity
Some easy and some hard
46
47
Jira Evaluation Summary
Using Both
Product Perspec7ve and the User Perspec7ve
• If we find some aZribute with low
performance from the user perspec7ve
• Then we can go back to the product
perspec7ve, figure out why, and improve
48
49
JIRA Evaluation by Attribute
Excerpt
For example:Login BuZon Visibility
(product usability aZribute)
50
Using Ques7onnaires
The process:
1. Use your model of what you want to measure and improve
2. Design the ques7onnaire according to the model
3. Execute the ques7onnaire
4. Analysis
One UX Model
Ques7onnaires Used
SUMI (So1ware Usability Measurement Inventory )
QUIS (Ques:onnaire for User Interface Sa:sfac:on )
PSSUQ (Post-‐Study System Usability Ques:onnaire )
ASQ (A1er-‐Scenario Ques:onnaire )
PUTQ (Purdue Usability Tes:ng Ques:onnaire )
PHUE (Prac:cal Heuris:cs for Usability Evalua:on )
SUS (System Usability Scale )
IUI (Isometrics Usability Inventory)
54
55
What do you do with all this?
56
Itera7ve Improvement
• Measure external quality – usability with
heuris7cs
• Measure in-‐use usability with logging or
ques7onnaires
• Go back to external and improve
Goal is to Understand, Evaluate and Improve
58
Next Steps
• Produce an ac7on plan
– What usability aZributes are important to your
organiza7on?
• Develop a model
– What data can you collect/Which technique can
you use
• Maybe some elements of the model drop out-‐can’t be
measured that easily
• Start collec7ng and developing benchmarks
Conclusion
• Usability and UX are both abstract concepts
– Cri7cal component of quality
• Defining is different for each organiza7on
• Need a model for your organiza7on
• The model is the founda7on of what to
measure
• Once you can measure, then you can evaluate
and improve
Resources
61
These are all journal publications that can be accessed via these links…
Thanks
Ques7ons and Answers
Please fill out an evalua:on form and drop it in the
collec:on basket located at the back of the room.
Philip Lew
@xbosoE
phone: 408-‐350-‐0508
www.xbosoE.com