Embed Size (px)
DESCRIPTION
Presented at Software Test Professionals, New Orleans 2012;Today’s web-based applications (WebApps) containing complex business logic and which are sometimes critical to operating the business, now must have an increased focus on usability as well as the newer and broader term, user experience. Especially with SaaS based business models where users can switch applications at a heartbeat and pay by the month (or even a free trial), both usability and user experience become paramount as there is no up-front investment forcing a user to stay. The ISO 25010 standard describes a model for general usage in specifying and evaluating software quality requirements, one of which is usability, but there are no models or formal definitions for user experience, nor its relation to usability. For usability, ISO 25010 is intended as a general guideline to be adapted based on a specific context and lacks implementation specifics. UX, on the other hand, does not have any formal standard definition, although some models have been developed regarding its elements. Models and research have been used mostly for the purpose of understanding, rather than evaluating improvements. In this session, we draw relationships between usability and user experience and explore measurement and evaluation methods that can be used as the first step toward improvement.
Citation preview
Evaluating and Improving
Software Usability 902 : Thursday, 9:30am - 10:45am
Philip Lew
www.xbosoft.com
Understand, Evaluate and Improve
2
Agenda
• Introduc7on – Importance of usability – What is usability? – What is UX?
• Usability Modeling and Measurements • Case studies on measuring and improving • Summary
What is Quality?
Pertaining to soEware J
4
Requirements…
Importance of Usability
• Usability is important especially for web applica7ons – And now especially mobile applica7ons
• Saas, mobile are in alignment with and soon becoming equivalent to cloud • Most prevalent implementa7on method in the next 5 years
• Without good usability: – Users will leave the applica7ons – For mobile, if they can’t learn in 30 seconds, they won’t come back
Why is Usability Important?
Web and Mobile have Changed the Terrain
• Business models have changed – Instead of paying upfront and ‘owning’ the soEware
– Pay as you go, pay by subscrip7on
• Behavior and expecta7ons have changed
What is Usability
In rela7on to quality In rela7on to user experience
When you think Usability…
• Naviga7on • Efficiency • Responsiveness-‐performance • Learnability • AZrac7veness • Understandability
Anything else come to
mind?
Current Research – Usability
Quality
Usability Satisfaction
ISO 9126-1
ISO 9241-11
Usability Design Experts
ISO 25010
User Experience
Product Owners
Usability -‐ Key Characteris7c of Product Quality
Source: ISO 25010
Quality in Use ISO 25010
Notes on Sa7sfac7on and Usability
• Sa7sfac7on is a subjec7ve feeling dependent on many things other than usability: – A user can be highly
sa7sfied but the applica7on with low usability.
– An applica7on can be highly usable (high usability) but the user is not sa7sfied!
Highly usable software
My password doesn’t work
don’t have what I want I’m unsatisfied
I’m satisfied!
! low usability software
Finished my work today
Nice weather today
Usability-‐Major Component of Quality Quality and Usability in the so1ware development lifecycle
• Can be measured from the design point of view or of the product
• Can be measured ‘in-‐use’ with real users
Usability
Usability
Evolu7on of SoEware quality
Software processes Code
Process quality
Software Quality
(internal)
Software Quality
(external)
Software Quality (in use)
Product ? CMMI
assessment model
white box testing
black box testing ?
CMMI Assessment Company
Programmer Tester End User
Type of quality
What is measured
How measured?
Who measures?
ISO 9000 ISO 9241 ISO 9126 ISO 25010
Source: ISO 25010
What is Usability-‐Effect of the SoEware Product
Degree to which specified users can achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.
Usability in Actual Usage
• User role • Objec7ve • Task • Environment • Domain • …
specified users
specified goals
specified context of use
What else can you think of?
Usability UX, Usability and Quality in Use • Difficult to understand their relationships • Lack of consensus in meaning in literature • Not totally clear how related to quality • Standard definition for user experience is still not available
and User Experience
Some UX Defini7ons UX DESCRIPTION
D1 a continuous process of user engagement with the product [10] D2 entire set of affects that results in user-product interaction [11] D3 the evolution of usability [12] D4 elaboration of the satisfaction component of usability [13] D5 a categorization of “do-goals” (pragmatics) and “be-goals”(hedonics) [3], [14]
D6 infinite small experiences relating to people, products and contexts [6] D7 consequence of user’s and product’s characteristics when interacted in a
specific environment [1] D8 degree to which specified users can achieve actual usability, safety, and
satisfaction in use in a specified context of use [15] D9 A person's perceptions and responses that result from the use or anticipated use
of a product, system or service [16]
D10 the degree of positive or negative emotions that can be experienced by a specific user in a specific context during and after product use and that motivates for further usage [17]
User Experience (UX) • Pragmatic (do-goals) : refers to the product's
perceived ability to support the task achievement and focuses on the product’s utility and usability in completing tasks, the ‘do-goals’ of the user.
• Hedonic (be-goals) : product's ability to support the user’s achievement of ‘be-goals’, such as being satisfied, pleasure.
• Real goal of the end user is fulfilling be-goals such as being autonomous, competent, related to others, stimulated, etc.”
• Be-goals are the driver of UX Hassenzahl
21
Rela7ng Usability and User Experience
Quality characteris7c
508 compliancy Func7onality User Interface
Usability Learnability
Product aZributes
Text/Sound aZributes
Appropriate Func7ons Easy to use Easy to learn
UX Pragma7c Do Goals Effec7ve and Efficient
UX Hedonic Be Goals Sa7sfac7on (pleasure, sense of community…)
Usability Measurement Effec7veness in use Efficiency in use Learnability in use
Sa7sfac7on Measurement Pleasure Comfort Sense of
Community Trust
Context
USER EXPERIENCE
Now that we understand it ALL…
We Need to Define Usability For Our Own Purposes
22
Let’s Define Usability/UX From the Product Viewpoint
Usability
Save Loca7on
Characteris7c 1 Characteris7c 2 Characteris7c n
Subcharacteris7c 1 Subcharacteris7c 2 Subcharacteris7c n
AZribute 1 AZribute 2 AZribute n
Understandable Naviga7on
Control Stability Previous-‐Next
Home Loca7on Ease of finding
Defining Usability from an Effect-‐Real usage Point of View
Quality
Errors
Effec7veness Efficiency Characteris7c n
Accuracy Completeness Subcharacteris7c n
AZribute 2 AZribute 3 AZribute 1
Usability
Sa7sfac7on
Some Poten7al AZributes Measurements for ‘in use’ Usability
• Effec9veness – Comple7on rates – Error rate – Help usage
• Efficiency – Task 7me – Backtracking
• Learnability – Learning rate – Task 7me devia7on
Can you think of any others, par9cular to
your applica9on?
26
Adding in UX Sa9sfac9on in use • Degree to which users are sa7sfied in a specified context of use. Sa7sfac7on is the user’s response to interac7on with the product, including amtudes towards use of the product. Subdivided into sub-‐characteris7cs: – Likability (cogni7ve sa7sfac7on) – Pleasure (emo7onal sa7sfac7on) – Comfort (physical sa7sfac7on) – Trust
UX Component
External Quality Requirements (for Shopping Cart Entity) 1 Usability
1.1 Understandability
1.1.1 Icon/label ease to be recognized
1.1.2 Information grouping cohesiveness
1.2 Learnability
1.2.1 ………………………………………………………..
1.3 Ease of Use
1.3.1 Control permanence
1.3.2 Helpfulness
Example of Product Quality Model with AZributes (Operability in ISO 25010)
28
Learnability • Degree to which the soEware product enables users to
learn its applica7on -‐ 7 principles (ISO 9241-‐10) – Suitability for the task -‐ should be suitable for the user’s task and skill level
– Self-‐descrip7veness-‐ should be clear what the user should do next
– Controllability -‐ the user should be able to control the pace and sequence of the interac7on
– Conformity with user expecta7ons -‐ should be consistent
– Error tolerant -‐ forgiving – Suitability for individualiza7on -‐ should be able to be customized to suit the user
– Suitability for learning -‐ should support learning
29
Ease of use
• Degree to which the soEware product makes it easy for users to operate and control it. – controllability – error tolerance (by operator) – conformity with user expecta7ons
30
Ease of Use -‐ Helpfulness
• Degree to which the soEware product provides help when users need assistance including help that is: – easy to find – comprehensive – effec7ve
Example: Usability Measurement
31
ACribute
Scale How Calcula9on Goal Current
Help completeness
Percent of Menu items with help
Compare menus and help items
% 90% 40%
Ease of access
Keystrokes to find/use a feature/func7on/informa7on
Sample 50 items
average 3.5 12
Consistency Number loca7ons for same buZon
Examine menus and doc.
integer 1 5
What types of usability measures are these?
32
Using a measurement model
Measurements
Metrics
Evalua7on
Measurement Func7on
Indicator
Evalua7on Evalua7on
Sub-‐characteris7cs
Evalua7on Evalua7on Characteris7c
Usability
composed of
composed of
Usability-‐Review
• Can be measured from the design point of view or of the product
• Can be measured ‘in-‐use’ with real users
usability usability in use
UX
34
Measuring Usability
Methods and models Now that we know what it is, and what to measure, then HOW?
Measurable AZributes
• Usability characteris7c • Descrip7on and purpose • How to measure • What is measured • Measurement/Calcula7on • Range (min, max) • Metric • Objec7ve (goal) versus Current (indicator)
Once you have a model (what you are going to measure), then you start doing IT!
UX/Usability Measurement Methods
Let’s get started
Logging
Heuris9c Evalua9on
Observa9on Labs
Focus groups
Ques9onnaires
37
Measurement methods Expert Evalua9on (Heuris9c evalua9on) • Define a set of rules or criteria with measurements and evaluate against them
Web-‐based logs • Collect user ac:vity data
– Mistakes and errors – How long it takes to complete tasks – Comple:on rates
Ques9onnaires • Quan7ta7ve subjec7ve measurement of UX characteris7cs
Heuris7c Evalua7on
Using a checklist But a liZle more complicated than
yes/no
38
39
Heuristic Evaluation of a Pharmacy application
Each usability attribute has a quantifying metric. • For example, for Predic9ve textual aCribute, users should be able to understand a buCon’s results prior to pressing it.
• Direct metrics need to be designed, i.e.: – (0) No support at all – (1) Par9al – (2) Complete
• Need a mapping from 0, 1, 2 to something more understandable, i.e. 2 = 100, and 1 = 60
• Need an indicator to interpret the level of sa9sfac9on met with decision criteria with acceptability ranges in a percentage scale: – 0-‐40 (unsa9sfactory –red) means changes must take place with high
priority; – 40-‐70 (marginal –yellow) indicates a need for improvement ac9ons; – 70-‐100 indicates a sa9sfactory level –green-‐ for the analyzed
aCribute.
40
2.2.2 Error Recovery Support " In the current state, users filling a new prescription are
supported well in error recovery (2.2.2) in automatic cursor positioning
41
Heuristic Evaluation – Pharmacy Software The purpose of the evaluation • Understand the external quality level of the learnability sub-characteristic for
filling a new prescription
Table 2 - excerpt of the whole current evaluation
External Quality Requirements Measure EI value P/GI value Global Quality Indicator 61.97%
1 Usability 60.88% 1.1 Understandability 83% 1.1.1 Icon/label ease to be recognized 100% 1.1.2 Information grouping cohesiveness 66% 1.2 Learnability 51.97% 1.2.1 ……………………………………………… … 1.3 Operability 49.50% 1.3.1 Control permanence 100% 1.3.2 Expected behaviour 50% 2 Content Quality 63.05% 2.1 Content Suitability 63.05% 2.1.1 Basic Information Coverage 50% 2.1.1.1 Line item information completeness 2 50% 2.1.1.2 Product description appropriateness 50% 2.1.2 Coverage of other Contextual Information 76.89% 2.1.2.1 ……………………………………………….. … 2.1.2.2 Return policy information completeness 33%
Heuris7c Evalua7on of Usability-‐Shopping Cart
JIRA Heuristic Usability Evaluation
Usability Logging
Collec7ng user ac7vity and behavior Tradi7onally used for other things
like…
44
Usability Logging • Iden7fy users by using session ID to iden7fy a unique user.
• Itera7vely insert code into the applica7on
• Collect data • Analyze the data for each aZribute in different dimensions and aggrega7ons
• Determine the need for further calcula7ons and what aZributes to measure further
• Revise the data we are collec7ng, adding or decreasing granularity
Some easy and some hard
46
47
Jira Evaluation Summary
Using Both Product Perspec7ve and the User Perspec7ve
• If we find some aZribute with low performance from the user perspec7ve
• Then we can go back to the product perspec7ve, figure out why, and improve
48
49
JIRA Evaluation by Attribute Excerpt
For example:Login BuZon Visibility (product usability aZribute)
50
Using Ques7onnaires
The process: 1. Use your model of what you want to measure and improve
2. Design the ques7onnaire according to the model
3. Execute the ques7onnaire 4. Analysis
One UX Model
Ques7onnaires Used SUMI (So1ware Usability Measurement Inventory ) QUIS (Ques:onnaire for User Interface Sa:sfac:on ) PSSUQ (Post-‐Study System Usability Ques:onnaire ) ASQ (A1er-‐Scenario Ques:onnaire ) PUTQ (Purdue Usability Tes:ng Ques:onnaire ) PHUE (Prac:cal Heuris:cs for Usability Evalua:on ) SUS (System Usability Scale ) IUI (Isometrics Usability Inventory)
54
55
What do you do with all this?
56
Itera7ve Improvement
• Measure external quality – usability with heuris7cs
• Measure in-‐use usability with logging or ques7onnaires
• Go back to external and improve
Goal is to Understand, Evaluate and Improve
58
Next Steps
• Produce an ac7on plan – What usability aZributes are important to your organiza7on?
• Develop a model
– What data can you collect/Which technique can you use
• Maybe some elements of the model drop out-‐can’t be measured that easily
• Start collec7ng and developing benchmarks
Conclusion
• Usability and UX are both abstract concepts – Cri7cal component of quality
• Defining is different for each organiza7on • Need a model for your organiza7on • The model is the founda7on of what to measure
• Once you can measure, then you can evaluate and improve
Resources
61
These are all journal publications that can be accessed via these links…
Thanks Ques7ons and Answers
Please fill out an evalua:on form and drop it in the collec:on basket located at the back of the room.
Philip Lew
@xbosoE
[email protected] phone: 408-‐350-‐0508 www.xbosoE.com
