Upload
arline-clarke
View
220
Download
0
Tags:
Embed Size (px)
Citation preview
HCI460: Week 3 LectureHCI460: Week 3 LectureSeptember 23, 2009September 23, 2009
2© April 21, 2023
Project 1 recap
Q & A
Research overview
Defining usability– Usability measures
Testing with users
Preparation for a usability study
Assignment for next week
Outline
3
Project 1 Recap
© April 21, 2023
4© April 21, 2023
No deductions for late submissions but from now on:– In class students: Projects are due Wed 11:59 pm Central Time– DL students: Projects are due Sun 11:59 pm Central Time
Please submit through COL. – One person should submit for the entire group.
Feedback on Project 1a (Individual Notes)Project 1 Recap
5© April 21, 2023
Assignments differed along two dimensions:– Number of issues found
• Some found only a few, some found a lot.– Level of polish
• Some turned in polished deliverables, some turned in “quick and dirty” work.
Real world: perfection vs. efficiency
Recognize when “quick and dirty” is needed and when professional and final deliverable is needed.
At this stage of the project (individual notes), it was better to have more issues and present them in a “quick and dirty” way than to have fewer issues but a more polished deliverable.
Feedback on Project 1a (Individual Notes)Project 1 Recap
6© April 21, 2023
Project 1b is due today at midnight for in-class students and on Sunday at midnight for DL students.– One person from each group should upload it to COL.
What have you learned?
What was difficult / challenging?– About the evaluation itself– About evaluating as a group
How did you overcome the challenges?
What will you do differently next time?
Did you see any value in conducting the evaluation as a group?
Project 1b: DiscussionProject 1 Recap
7
Q & A
© April 21, 2023
8© April 21, 2023
Interaction has been excellent thus far
Clarification will always be provided, just ask
Exercises allow for alternative feedback
Feedback
Opportunities– Push the User eXperience (UX) envelope– Professional development
Class Interaction ~ OpportunityQ & A
9© April 21, 2023
UXalliance.com
mobileHCI– Eating usability– Touch interface debate
Conference RecapQ & A
10
Research Overview
© April 21, 2023
11© April 21, 2023
What do we do?
What do you want to do?
Why Are You Here?Research Overview
12© April 21, 2023
Research background
Design background
Paths Into UX ResearchResearch Overview
13© April 21, 2023
If someone asked, what is it?
What we strive to do is answer questions.
Methodologies and techniques are just tools.
As practitioners, we must be focused on the questions.
Often, finding the answer will require much deeper and complex methodologies than discount usability testing will provide OR NOT!
Let’s explore and quite possibly push the boundaries of user experience research tactics.
Thinking About UX ResearchResearch Overview
14
May have mentioned….
Web sites Software applications Interactive Voice Response systems (IVRs) Speech recognition systems Text-to-speech (TTS) interfaces Voice mail systems Unified messaging interfaces Internet applications Games and gaming systems Telecommunication products and services Call center applications Consumer products Wireless devices Packaging
What Can’t We Evaluate?Research Overview
© April 21, 2023
Consider an iPhone…
15
What Can We Test?Research Overview
© April 21, 2023
Three Dimensions of Feedback
16
How people evaluate objects− Commercials− Packages− Online advertisements− Products− …
Research Overview
© April 21, 2023
Attitude What users “say” ...
Influencers– Social status– Emotion– Coolness / Hip
Reveal– Feature importance – Purchase intent
17
Research Overview
© April 21, 2023
Behavior What users actually “do”
Ultimately, behaviors are what we wish to shape
Give users context, a task and stimuli and then – Observe what users do
Behavior drives usage
18
Research Overview
© April 21, 2023
Attention What users “focus” on…
What happens inside the head
Sometimes users are unaware
Often attention measured by
eye tracking
19
Research Overview
© April 21, 2023
20© April 21, 2023
Defining Usability
21© April 21, 2023
What do we do?
What do you want to do?
Why Are You Here?Defining Usability
22© April 21, 2023
We want to make things better.
But how do you measure “better?”
Can usability be defined?
Can Usability Be Defined?Defining Usability
23© April 21, 2023
The UX HoneycombDefining Usability
24© April 21, 2023
Lights are on.
Turn off the light.
What Makes Something Usable?Defining Usability
25© April 21, 2023
Turn the knob to move the arrow to the right.
Are There Universal Rules or Stereotypes?Defining Usability
26© April 21, 2023
Make the shower temperature “just right.”
Interaction Not Always UniversalDefining Usability
27© April 21, 2023
How About More Complicated Interfaces?Defining Usability
28© April 21, 2023
Can Interfaces Be Learned?Defining Usability
29© April 21, 2023
Emphasis on Usability in Many FieldsDefining Usability
30© April 21, 2023
No measures exist.
Why?
What can we measure?– Success/Fail– Time– Accuracy– Satisfaction– ...
Naysayers: You Cannot Measure UsabilityDefining Usability
31© April 21, 2023
From UPA:– Usability is an approach to product development that
incorporates direct user feedback throughout the development cycle in order to reduce cost and create products and tools that meet user needs.
From Krug’s “Don’t Make Me Think:”– Usability really means making sure that something works well…
so that a person of average ability and experience can use the thing for its intended purpose without being hopelessly frustrated.
Definitions of UsabilityDefining Usability
32© April 21, 2023
From ISO 9241-11:– The extent to which a product can be used by specified users to
achieve specific goals with effectiveness, efficiency, and satisfaction in a specific context of use.
• Effectiveness: Being able to complete the task• Efficiency: Amount of effort required to complete task• Satisfaction: Degree of happiness or fulfillment while
performing task (or rather the absence of pain and frustration)
Definitions of UsabilityDefining Usability
33© April 21, 2023
There is just too much going on to get reliable data—NOT!
Measurement must:– Be observable– Be quantifiable– Have sound control– Have understanding of context
Naysayers: Usability Data Are Too NoisyDefining Usability
34© April 21, 2023
Not All Measures Are Created EqualDefining Usability
Bad Inappropriate Good
35© April 21, 2023
Performance metrics
Issues-based metrics
Self-reported metrics
Behavioral and physiological metrics
Combined and comparative metrics
Types of Usability MetricsDefining Usability
36© April 21, 2023
Task success– Binary: Success/fail– Levels of success: Complete success/partial success/…/failure
Time on task– Actual time– Thresholds
Errors– Actual number
Efficiency– Deviations from optimal path– Lostness (Smith, 1996)– Combination of task success and time (NIST, 2001)
Learnability– Metrics above taken from trials within same session, with breaks, between
sessions
Performance MetricsDefining Usability
37© April 21, 2023
Usability issues– Number of issues found– Percentage of participants who found
an issue
Severity ratings– High severity– Medium severity– Low severity
Issues-Based MetricsDefining Usability
59September 23, 2009
Usability Issue: The Rapid Part Add feature does not recognize part numbers if dashes are not entered. The items are added to the cart but they are not recognized.Recommendation: The search functionality should be able to handle part numbers without dashes. If it can’t, instructions should say that the part number has to include dashes. Alternatively, provide three input fields, separated by dashes. Automatically advance to the next field if maximum number of characters have been entered.
Usability Issue: The Rapid Part Add gets visually lost among all the other elements on the page and may be difficult to notice.Recommendation: Put the Rapid Part Add in a shaded box to make it stand out.
Good Practice: Rapid Part Add allows users to quickly enter their shopping list if they know the part numbers of the items without having to search for the items one by one. Also, users’ own part numbers are accepted in the Rapid Part Add tool, which is helpful.
Usability Issue: There is no error check when adding parts through Rapid Part Add. If users make an error in the part number, they will not have an opportunity to correct it.Recommendation: Provide users with feedback If the part number is not recognized. H
L
M
Presenting Results
Example
HIGH
MEDIUM
LOW
38© April 21, 2023
Post-task ratings– Likert
Post-session ratings– Likert
System Usability Scale (SUS) (Brook 1996)– Likert
NASA TLX– Likert
Specific attribute– Agreement
Answers to open-ended questions
Self-Reported MetricsDefining Usability
39© April 21, 2023
System Usability Scale (SUS)Defining Usability
40© April 21, 2023
Verbal and non-verbal– Positive / negative comments
Facial expressions– Computerized analysis
Eye movements– Attention
Pupil diameter– Workload
Skin conductance and heart rate– Stress
Behavioral and Physiological MetricsDefining Usability
41© April 21, 2023
Usability scorecards– Comparison– Harvey balls
– Radar charts
Combined and Comparative MetricsDefining Usability
42© April 21, 2023
Again, Metric Selection is Non-TrivialDefining Usability
Bad Inappropriate Good
43© April 21, 2023
Consider a Horse Race: Which Measure is Good?Defining Usability
44© April 21, 2023
Different Conditions: Yes, That is SnowDefining Usability
45© April 21, 2023
Appropriate Metrics Depend on Many FactorsDefining Usability
46© April 21, 2023
Objective: – Tons of user complaints that volume is too soft.– We have three new versions and want to know if the problem has
been fixed.
Measure: – Rating of the perceived sound level
1 2 3 4 5 6 7
Exercise: Measurement ScaleDefining Usability
47© April 21, 2023
Evaluating UsabilityDefining Usability
48© April 21, 2023
Testing with Users
49© April 21, 2023
Inspection methods = no users, only UX experts– Heuristic evaluation– Expert evaluation– Competitive evaluation– Cognitive walkthrough– Pluralistic walkthrough
Methods involving users– User testing (lab, longitudinal)– Eye tracking– Focus groups– Surveys– Ethnographic research
Methods with No Users vs. Methods with UsersTesting with Users
50© April 21, 2023
Artificial situations
Results cannot prove product will work
Participants not necessarily actual target market
Might not be the right thing to do
Others? (Beyond Rubin?)
Sample size considerations
Feature coverage
Limitations of Testing (Rubin … Lab Testing)
Testing with Users
51© April 21, 2023
Walk Up And UseTesting with Users
52© April 21, 2023
FrustrationsTesting with Users
53© April 21, 2023
Test Interval Long Enough?Testing with Users
54© April 21, 2023
Hypothesis testing
Participants are randomly chosen
Tight controls in place
Control groups employed
Sample size sufficiency
UX Research = Experimental Method (Rubin) Testing with Users
55© April 21, 2023
Study Preparation
56© April 21, 2023
Understand study objectives
Determine the type of test
Identify target users
Understand the stimuli
Create the test plan
Develop the screener
Determine analysis and expect results
Develop the moderator’s guide – next week
Test PlanningPreparation for a Usability Study
57© April 21, 2023
Identifying the research objectives will help you select appropriate approach and methodology for the study.
Ask stakeholders:– What are you hoping to learn from this study?– What decisions will you need to make based on the outcome of
the study?
Understand Research ObjectivesPreparation for a Usability Study
58© April 21, 2023
Is it a formative or summative test?
Determine the Type of TestPreparation for a Usability Study
59© April 21, 2023
Earlier in product life cycle
Diagnostic
Qualitative
Study goal: Determine key strengths and weaknesses of the system
Ultimate goal: Improve the user experience
E.g., “We want to improve the website we are developing, so that the users can easily find what they are looking for.”
Determine the Type of Test: FormativePreparation for a Usability Study
60© April 21, 2023
Later in product life cycle
Verification; quantitative
Study goal: Determine how the system compares to usability standards, benchmarks or competitors
Ultimate goal: To make sure the product is ready for launch (can involve making legally defensible statistical, marketing, or regulation claims)
E.g., “We need to be sure that the new drug label will not increase the number of dispensing errors.”
Results should often be statistically valid.
Determine the Type of Test: SummativePreparation for a Usability Study
61© April 21, 2023
Difference in terminology between many professionals and Rubin:– Formative tests: Include Rubin’s exploratory and
assessment tests (1st edition)– Summative tests: Include Rubin’s validation and
comparison tests (1st edition)
In his 2nd edition, Rubin calls exploratory tests “formative” and assessment tests “summative.”– This is not a typical classification and we will use this one
Project 2: Formative study
Project 3: Summative study
Determine the Type of TestPreparation for a Usability Study
62© April 21, 2023
Definitions
Benefits
Costs
Between vs. Within Groups DesignPreparation for a Usability Study
63© April 21, 2023
Define the target user groups that should be represented in the study.
Identify Target UsersPreparation for a Usability Study
64© April 21, 2023
Become familiar with the product or interface of interest and understand its capabilities and constraints.
Explore more than just the product itself = understand the whole experience:– Look at the packaging.– Read the manual and other
accompanying materials.
If the stimulus is a prototype:– Make sure it works.– Suggest quick fixes prior to
testing if there are glaring usability issues.
Review the StimuliPreparation for a Usability Study
65© April 21, 2023
A test plan is a document that helps:– Organize all information about a study. – Communicate this information to the research team and study
stakeholders.
Test plan should be clear and detailed.– Bulleted lists and diagrams help convey important details in a
clear manner.
Create the Test PlanPreparation for a Usability Study
66© April 21, 2023
Objectives of the study broken up into specific questions that the study has been designed to answer.
Description of the stimuli, including version or model numbers where appropriate.
Description of the participants with an indication of strict requirements and allowable variances.
A Good Test Plan Contains…Preparation for a Usability Study
67© April 21, 2023
Description of the methodology, including:– The procedure (e.g., use
of a think-aloud protocol and within- or between-groups study design)
– Measures used (e.g., time on task, ratings)
A Good Test Plan Contains…Preparation for a Usability Study
68© April 21, 2023
Requirements for equipment needed for testing (e.g., size of TV or computer screen, computer operating system, and types of cameras – face shot, desktop shot etc.).
Pictures/sketches or description of the lab setup denoting how the stimuli should be arranged, where the moderator and participant should sit, and where the camera(s) should be set up. – Esp. beneficial if complex
setup with multiple elements.
A Good Test Plan Contains…Preparation for a Usability Study
69© April 21, 2023
The project logistics, including:– Test dates and times– Test location(s)– Deliverable deadlines for
the stakeholders and the research team
– Etc.
A Good Test Plan Contains…Preparation for a Usability Study
70© April 21, 2023
… and iterate it based on their feedback.
Have Stakeholders Review the Test PlanPreparation for a Usability Study
71© April 21, 2023
The Screener (or Screening Questionnaire) is a document according to which the study participants will be recruited.
Examples of screener elements?
Internal recruit vs. external recruit
Tasks of the recruiter– Screen– Schedule– Follow-up
List vs. database recruit
Develop the ScreenerPreparation for a Usability Study
72© April 21, 2023
Bad participants
No data
Risks of Poorly Developed ScreenerPreparation for a Usability Study
73© April 21, 2023
Poor show rate
Bad participants
No data
Risks of Bad RecruiterPreparation for a Usability Study
What Makes A Screener Effective? It is clear for recruiters, clients, and respondents to understand.
– Participant should not have to ask “what do you mean?” – Questions should be able to be answered with minimal thought
and quickly.• The more prospective participants they “think about it” – the
more likely they will overthink it and not give an honest response.
– It should be easy enough for respondents to accurately remember details to answer questions.
– Summarize quotas, termination criteria, and nice-to-haves in a separate section.
Preparation for a Usability Study
© April 21, 2023 74
What Makes A Screener Effective? Not overly restrictive
– Strive for the least restrictive screener that will get the respondents you need to satisfy test objectives. This will reduce recruiting costs.
– Nested recruits make recruits restrictive.– Recruits of users of specific features or devices tend to be more
restrictive.
Provides directions to the facility.
Require tangible artifacts– State-issued ID– Related products (phone, medicine box, etc.)
Preparation for a Usability Study
© April 21, 2023 75
What Makes A Screener Effective? Efficient
– Aim for max 20 questions.– Minimum number of questions to effectively screen respondents.
Contains all the details– Reminder to recruit talkative participants– Phone numbers on the day of testing and before testing– Let participants know they will be videotaped– Under 16 consent form requirements
Enough soak time - appropriate for the difficulty of the recruit– Standard soak time of 2 weeks?– We should create a list of specific types of studies that need
long soak times
Preparation for a Usability Study
© April 21, 2023 76
77© April 21, 2023
Avoid yes/no questions unless absolutely necessary.– They reveal a pattern and can be considered leading.
• We want honest, knee jerk responses that reflect current behavior.
Avoid questions that allow one to figure out what you’re looking for and answer accordingly.– Self-identification questions – Have participants state the activities that they perform, instead of
having the recruiter read a list.
Questions To AvoidPreparation for a Usability Study
Test Plan Creation Detailed, clear, and explicit test plan is the key to a successful
global study.– It improves communication between teams.– It sets expectations.– It links the methods and procedures to study objectives.
Preparation for a Usability Study
© April 21, 2023 78
79© April 21, 2023
Control what you can control.
Lesson Learned: Anticipate outcomes and their implications.– Spend time to think hard about research.– Effort spent early will make the study better and might prevent
another study.
Ready, Fire, Aim…?!?!Preparation for a Usability Study
80© April 21, 2023
Logistics
Facility
Moderator’s Guide
Other StepsPreparation for a Usability Study
81
Assignment for Next Week
© April 21, 2023
82© April 21, 2023
As a group, select a product / interface to test, for example:– Application on a mobile phone– Medical device (e.g., blood pressure monitor, glucose meter)– Small home appliance (e.g., coffee maker)– Instructions to a game / puzzle– Utility bill– Software application or a website (only if you must)
Bring it with you to class next week and on Oct 7th & 14th.
Project components:– 2a: Test Plan – Due Next Week– 2b: Screener, Moderator’s Guide– 2c: Data collection– 2d: Final Report
Project 2: Formative Usability StudyAssignment for Next Week
83© April 21, 2023
Develop a test plan for testing your chosen product / interface.
The test plan should describe the following:– Objectives of the study– Stimuli– Participants– Procedure– Measures – Equipment and Lab Setup– Logistics
Project 2a: Test Plan – Due Next WeekAssignment for Next Week
84© April 21, 2023
Grading criteria:– Are all the components included in the test plan? (see previous
slide)– Are the objectives realistic?– Are the participants described in enough detail?– Does the methodology (including measures) appropriately
address the objectives of the study?– Overall, is the test plan clear and precise?– Is the test plan well structured and laid out?
• Proper alignment• Consistency of fonts, sizes, spacing, phrasing etc.
Project 2a: Test Plan – Due Next WeekAssignment for Next Week
85© April 21, 2023
Reading for Next Week Handbook of Usability Testing by Rubin:
– Chapter 4: Skills for Test Moderators– Chapter 7: Find and Select Participants– Chapter 8: Prepare Test Materials
Assignment for Next Week
86© April 21, 2023
Recap of To-Dos for Next Week As a group, select a product / interface to test.
– Bring it to class on Sep 30th.
Develop the test plan for formative usability testing of the selected product / stimulus.– Submit through COL by midnight Sep 30th (in-class students) or
Oct 4th (DL students).
Do your reading.
Assignment for Next Week