Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Usability TestingSubjective Questionnaires
Cristina Cachero
This project has been funded with support from the European Commission under the Lifelong Learning Programme
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Inquiry method: you ask your users about what they do, but you do not observe it directly.
There are several proposals that have shown enough internal and external validity, and that can be applied at different times during the performance test
After all the tasks have been finished (test-level questionnaires)
Before/after each task is performed (task-level questionnaires)
Also, questionnaires may include open questions that can be codified and analyzed.
This project has been funded with support from the European Commission under the Lifelong Learning Programme
2
Subjective Questionnaires
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Test-level (post-test) questionnaires (1/2): CSUQ: Computer System Usability Questionnaire [Lewis 1995]: 19 sentences,
all positive. They measure four dimensions: system utility, information quality, interface quality and general satisfaction. It was devised to be administered off-line
PSSUQ: Post-Study System Usability Questionnaire: variation of CSUQ to be administered in person
QUIS: Questionnaire for User Interface Satisfaction. 27 sentences divided in five groups: general reaction, screen, terminology/system info, learning, system capabilities.
EUCS : End-user computing satisfaction [Abdinnour-Helm 2006]. 12 items representing five dimensions: content, accuracy, format, ease of use and timeliness. All the items are combined into a global EUCS measure. 5-point scale (Almost never… almost always)
USE: Usefulness, Satisfaction and Ease of Use [Lund 2001]. 30 items divided in four categories: utility, satisfaction, ease of use and ease of learning. For each one, seven-point scale items..
This project has been funded with support from the European Commission under the Lifelong Learning Programme
3
Subjective Questionnaires
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Test-level (post-test) questionnaires (2/2): SUS: Systems Usability Scale [Brooke 1996]: 10
questions, 5 formulated in positive and 5 formulated in negative. It gives a global measure of the site usability
Net promoter Scores (NPS) ©: A single score regarding the fidelity of the user
SUMI [Kirakowski 1996] Questionnaires that rank sites (based on proprietary DB)
SUPR-Q: UX questionnaire . Includes usability (4 items), credibility (trust, value & comfort, 5 items), loyalty (2 items, one of which is the Net Promoter Score) and Appearance (2 items). Five-point scale from strongly disagree to strongly agree except for Item 10 (net promoter question)
WAMMI, ACSI, Opinion Lab
Proprietary (beware of reliability and validity!!)
This project has been funded with support from the European Commission under the Lifelong Learning Programme
4
Subjective Questionnaires
Source: ISO 13407
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
5
Post-test SQ: EUCS [Abdinnour-Helm 2006].
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
6
Post-test SQ: USE [Lund 2001]
http://hcibib.org/perlman/question.cgi?form=USE
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
SUS (Systems Usability Scale) [Brooke 1996] 10 questions, 5 formulated in positive and 5 formulated in negative. It gives a global measure of the site usability (it does not distinguish among different components). It is the most reliable [Tullis and Stetson, 2004]
This project has been funded with support from the European Commission under the Lifelong Learning Programme
7
Post-test SQ: SUS [Brooke 1996]
The mean SUS value is 66%, the 25 percentile is 57% and the 75 percentile is 77%. This means that we should get a SUS greater than 80% (taking into account confidence intervals) to be reasonably sure of the global satisfaction of our users.
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Activity: Read the original SUS paper, published by Brooke in 1996. You have the paper available in the Moodle platform.
This project has been funded with support from the European Commission under the Lifelong Learning Programme
8
Post-test SQ: SUS
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
9
Post-test SQ: SUS: Exercise
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use 4. I think that I would need the support of a technical person to be able to use this system
5. I found the various functions in this system were well integrated
6. I thought there was too much inconsistency in this system
7. I would imagine that most people would learn to use this system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get going with this system
Strongly Strongly Disagree Agree
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
10
Post-test SQ: SUS: Score?
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use 4. I think that I would need the support of a technical person to be able to use this system
5. I found the various functions in this system were well integrated
6. I thought there was too much inconsistency in this system
7. I would imagine that most people would learn to use this system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get going with this system
Strongly Strongly Disagree Agree
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
11
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use 4. I think that I would need the support of a technical person to be able to use this system
5. I found the various functions in this system were well integrated
6. I thought there was too much inconsistency in this system
7. I would imagine that most people would learn to use this system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get going with this system
Strongly Strongly Disagree Agree
Post-test SQ: SUS. Steps Reverse items 2,4,6,8,10 Sum positions for each item (0 to 4) Multiply by 2.5
4
1
1
4
1
2
1
1
4
3
TOTAL SCORE: 22
SUS SCORE: 22*2.5=55
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
12
Post-test SQ: SUS. Threshold value
Which should be our SUS objective?
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
13
Post-test SQ: SUS. Adaptation
The SUS vocabulary can be adapted to the idyosyncrasy of the particular system
Example: OHIM: system->web site
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
14
Post-test SQ: SUS. Calculator
Calculator: http://www.measuringux.com/SUS-scores.xls
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Net Promoter Scores: measure obtained through a single question (loyalty of the user to the app):
How likely is it that you’ll recommend this product to a friend or colleague? (0..10)
Three segments: Promoters: Responses from 9 to 10 Passives: Responses from 7 to 8 Detractors: Responses from 0 to 6
Promoter score (-100..+100): %Promoters-%Detractors
This project has been funded with support from the European Commission under the Lifelong Learning Programme
15
Post-test SQ: Net Promoter Score
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Questionnaires that provide rankings to compare your results against the results from other similar apps
SUPR-Q (Sauro) WAMMI (www.wammi.org) (SUMI
successor) ACSI (www.TheACSI.org): particularly
interesting form government websites OpinionLab (www.OpinionLab.com)
This project has been funded with support from the European Commission under the Lifelong Learning Programme
16
Post-test SQ: Rankings
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
17
SQ: Rankings. SUPR-Q (4 sub-scales)
NPS
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
18
Post-test SQ: Rankings. SUPR-Q. SUPR-Q Score: Sum all items + ½ item 10
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
19
Post-test SQ: Rankings. SUPR-Q. Besides the values for each factor, SUPR-Q is backed on a proprietary DB that allows to generate a ranking of percentiles
E.g. your app belongs to percentil 75 regarding usability, what means, it is among the best 25%. The SUPR-Q usability factor has a strong correlation with a SUS score, r = .96. p < .001, meaning just four questions account for 93 percent
of the variation in SUS (.96 squared). These questions are a good substitute for SUS on websites Validity and reliability: http://www.suprq.com/
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
20
Post-test SQ: Rankings. WAMMI
http://www.wammi.com/samples/index.html Results are divided into five areas:
attractiveness, controllability, efficiency, helpfulness and Learnability, plus an overall usability score.
The scores are standardized (from comparison to their reference database), so 50 is medium and 100 is perfect.
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
This project has been funded with support from the European Commission under the Lifelong Learning Programme
21
Post-test SQ: Rankings. WAMMI
Sample graphic showing how a give website positions in reference to average scores in each axis
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Circle five words that describe what you think about this design:
What are the three things you like best and least
about the Web site? _____________________________________________ ______________________________________________ ______________________________________________
This project has been funded with support from the European Commission under the Lifelong Learning Programme
22
Post-test SQ: proprietary (OHIM 1/2)
CuteStableResponsiveFriendlyHelpful
ReputableApproachableReliableClutteredGood
ConfidentTrustworthyCurrent and coolService orientedBoring
Easy to useConfusingComfortableAnnoyingInformativeOut-of-date
High-techSensitiveSecureStraightforwardAmateurish
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
If you could make one significant change to this Web site, what change would you make?
(Ask the participant if he is using/knows other IP websites) How do you find the site in comparison to other IP web sites?
Do you feel this site is current? Why? If you were to describe this site to a colleague in a sentence or
two, what would you say? Do you use the current OHIM web site? If Yes then Do you
think that the new OHIM web site clarity of structure is: Better, Same As, Worse or Don’t know, than the current one?
Do you have any other questions or comments about the Web site or your experiences with it?
This project has been funded with support from the European Commission under the Lifelong Learning Programme
23
Post-test SQ: proprietary (OHIM 2/2)
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Expectation Rating After Scenario Questionnaire (ASQ) Awareness-Usefulness gap
This project has been funded with support from the European Commission under the Lifelong Learning Programme
24
Task level (Post-Scenario) SQ
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Pre-task: How easy/difficult do you thing that the following task is going to be?
Post-task: How easy/difficult has been for you to carry out this task?
This project has been funded with support from the European Commission under the Lifelong Learning Programme
25
Post-Scenario SQ: Expectation Rating
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Three post-task questions: I feel satisfied with the easiness with which I have
completed the task. I feel satisfied with the time that it has taken me
to complete this task. I feel satisfied with the support information (online
help, messages, documentation, and so on) that I have had available while I was completing this task (only if your system offers online help)
This project has been funded with support from the European Commission under the Lifelong Learning Programme
26
Post-Scenario SQ: ASQ
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Two post-task questions: ‘‘Were you aware of this functionality prior to this
study? (yes or no) On a 1 to 5 scale, how useful is this functionality to
you? Calculation:
Convert the likert item to a binary scale (e.g. 4 or 5 useful, rest not useful) and draw the graph.
Differences between awareness and usefulness indicate which tasks to promote (e.g. by redesigning the visual hierarchy) in your app.
This project has been funded with support from the European Commission under the Lifelong Learning Programme
27
Post-Scenario SQ: Awar-Usefuln Gap
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Graph sample:
This project has been funded with support from the European Commission under the Lifelong Learning Programme
28
Post-Scenario SQ: Awar-Usefuln Gap
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
More questionnaires are appearing by the day in the literature. Many of them have been deemed necessary to cover the idiosyncrasy of new platforms/domains E.g. MPUQ: Mobile Phone Usability Questionnaire
This project has been funded with support from the European Commission under the Lifelong Learning Programme
29
SQ: Other questionnaires
Reliability and Validity of the Mobile Phone Usability Questionnaire (MPUQ)
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Can be post-test or post-task Reasons why the users are promoters or detractors of your
product Insights from users gathered from field studies Complaints about a product sent to the customer service Why the task was difficult to complete …
The way of processing this kind of questions is to turn the open comments into categories, quantify them and analyze them. http://www.measuringusability.com/blog/quantify-comments.php
This project has been funded with support from the European Commission under the Lifelong Learning Programme
30
SQ: Open Questions
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
All the questionnaires must be validated with respect to their reliability, validity and utility. Once validated, they still need to be revalidated when the context of application varies.
This project has been funded with support from the European Commission under the Lifelong Learning Programme
31
SQ: Validation
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
Lewis95] Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7, 57–78.
[Lund 2001] Lund, A.M. (2001) Measuring Usability with the USE Questionnaire. STC Usability SIG Newsletter, 8:2
[Chin 88] Chin, J. P., Diehl, V. A., and Norman, K. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. In: CHI '88. Conference Proceedings on Human Factors in Computing Systems. New York: ACM, pp. 213–218.
[Brooke 1996] Brooke, J. (1996). SUS: A “quick and dirty” usability scale, In Jordan, P. W., Thomas, B. T., and Weerdmeester, B. A. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis, pp. 189–194.
[Kirakowski 1996] Kirakowski, J. (1996). The software usability measurement inventory: Background and usage. In Jordan, P., Thomas, B., and Weerdmeester, B. (Eds.), Usability Evaluation in Industry. UK: Taylor and Francis, pp. 169–177.
[Lewis 1991] Lewis, J. R. (1991). Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ. SIGCHI Bulletin, 23(1), 78–81.
[Abdinnour-Helm 2006]. Using the End-User Computing Satisfaction (EUCS) Instrument to Measure Satisfaction with a Web Site.
[Tullis 2008] Thomas Tullis and , William Albert (2008). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics (Interactive Technologies). Morgan Kauffman.
This project has been funded with support from the European Commission under the Lifelong Learning Programme
32
References
Leonardo da Vinci Partnerships ProjectGUI USABILITY AND ACCESSIBILITY:
EXCHANGING KNOWLEDGE AND EXPERIENCES
These slides are made available under the license Creative Commons Attribution-NonCommercial-NoDerivs CC BY-NC-ND. More information about license: http://creativecommons.org/licenses/by-nc-nd/3.0/.These slides were created under Leonardo da Vinci Partnerships Project 2012-1-PL1-LEO04-28181 GUI USABILITY AND ACCESSIBILITY: EXCHANGING KNOWLEDGE AND EXPERIENCES (http://usability-accessibility.org/).
This project has been funded with support from the European Commission under the Lifelong Learning Programme
33
Attributions