View
220
Download
0
Category
Tags:
Preview:
Citation preview
Jesper Kjeldskov & Jan StageDepartment of Computer Science
Aalborg UniversityDenmark
New Techniques for Usability Evaluation of Mobile Systems
2Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Background
Mobile technologies and systems… PDAs, wearables, mobile phones, tablet computers
… challenge usability testing methods Users are physically mobile during use Use involve activities in physical surroundings Use context can be difficult to recreate realistically
This paper explore new techniques for usability testing mobile systems in laboratory settings
3Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Two Evaluation Approaches
Field experiments Realistic use context Difficult to control Complicated data collection Complex and time consuming Safety and ethical issues
Laboratory experiments Experimental control
High quality data collection
Lack of realism
4Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Laboratory vs. Field
Most usability evaluations of mobile systems are currently conducted in laboratory settings
A recent literature study revealed that… 41% of mobile HCI research involve evaluation 71% of this is done in laboratory settings
It is a widely adopted point of view that mobile systems require field evaluations, but… It is difficult to conduct field evaluations The added value of testing in the field is unknown Additional problems come at a high cost (time & effort)
5Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Overall Research Question
How can new techniques for usability tests of mobile systems increase realism of use in a laboratory setting?
… while facilitating systematic data collection in a controlled environment
Previous studies: Specialized use contexts This study: Physical mobility during use
6Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Two Experiments
Two experiments comparing techniques for lab- and field-based usability testing of mobile systems were conducted
Experiments explored different techniques requiring…1. Different levels of physical movement2. Divided cognitive attention
Example application: use of Short Message Service (SMS) on PDAs and mobile phones
Narrow focus on interaction rather than broad focus on use
7Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Five Laboratory Techniques…
Attention needed to navigate
None Conscious
BodyMotion
None 1. Sitting at a table or standing
n/a
Constant 2. Walking on a treadmill with
constant speed
4. Walking at constant speed on a
changing track
Varying 3. Walking on a treadmill with varying speed
5. Walking at varying speed on a
changing track
The experiment: measuring the relative strengths and weaknesses with reference to field testing
8Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
The Laboratory Experiments
5 conditions (6 test subjects per condition) Number of usability problems Performance (task completion time) Subjective workload (NASA TLX)
9Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
The Field Experiment
1 condition: walking in a pedestrian street (6 test subjects) Number of usability problems Performance (task completion time) Subjective workload (NASA TLX)
10Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Findings (1)
More problems found on average when seated at a table Statistical significance Lab techniques with physical movement comparable to
field evaluation
Mean number of usability problems identified by each technique
1. Sitting at a table
2. Walking on a treadmill with constant speed
3. Walking on a treadmill with varying speed
4. Walking at constant speed on a changing track
5. Walking at varying speed on a changing track
6. walking in a pedestrian street
11Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Findings (2)
Techniques Total
Lab 1 Lab 2 Lab 3 Lab 4 Lab 5 Field
Critical 4 4 3 4 3 3 4
Serious 11 11 9 9 9 8 17
Cosmetic 19 8 8 8 6 12 32
Total 34 23 20 21 18 23 53
No technique identified all problems Most problems found when seated at table (34) Comparable numbers of critical problems found (3-4)
Number of identified usability problems categorized by severity
12Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Findings (2)
Techniques Total
Lab 1 Lab 2 Lab 3 Lab 4 Lab 5 Field
Critical 4 4 3 4 3 3 4
Serious 11 11 9 9 9 8 17
Cosmetic 19 8 8 8 6 12 32
Total 34 23 20 21 18 23 53
More than double the number of cosmetic problems were found while seated compared to the other lab techniques
Number of identified usability problems categorized by severity
13Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Findings (2)
Techniques Total
Lab 1 Lab 2 Lab 3 Lab 4 Lab 5 Field
Critical 4 4 3 4 3 3 4
Serious 11 11 9 9 9 8 17
Cosmetic 19 8 8 8 6 12 32
Total 34 23 20 21 18 23 53
Only 3 out of 4 critical usability problems were identified on basis of the field technique
Number of identified usability problems categorized by severity
14Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Findings (3)
Techniques
Lab 1 Lab 2 Lab 3 Lab 4 Lab 5 Field
Mental demands 29 75 204 126 185 148
Physical demands 92 117 112 118 127 194
Effort 52 163 106 228 178 186
Overall workload 27 35 48 55 48 54
Sitting at a table (lab 1) required significantly less mental activity compared to all other techniques but lab 2
Overall, sitting or walking at constant speed is experienced significantly less demanding than any other technique
Subjective experience of workload with the different techniques
15Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Discussion (1)
Seating test subjects at a table… superior? Supported the identification of most usability problems Difference mostly accounted for by cosmetic problems
Less workload facilitated more thinking-aloud Relevance of cosmetic problems can be questioned
Increasing workload in lab Helped approximating the field condition but resulted in
fewer problems identified compared to when being seated
Better focus missing vital problems??
16Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Discussion (2)
Usability problems and mobility Physical motion in the lab triggered unique interaction
problems also found in the field Added value in relation to e.g. layout and button sizes
Data collection in the field All field tests were recorded with
a camcorder Difficult to capture good images
of screen The “bodyguard” effect Changing the role of the test monitors? Mounting small cameras on test subject and device?
17Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems
Future work
Cost/benefit of different techniques and settings Time and effort spent per problem found
When should lab or field tests ideally be applied?
How can field test techniques be improved? Improving data collection? Enforcing more experimental control? Doing something completely different?
Recommended