Upload
wcit-2014
View
243
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Workshop at the WCIT 2014 Human computer interaction, evaluation, usability testing, study design and methods. Peter Elkin, University at Buffalo
Citation preview
Human Computer Interaction Evaluation, usability testing,
study design and methods
Presented by: Peter L. Elkin, MD, MACP, FACMI
With Studies from EvaLAB
Marie-Catherine Beuscart-Zéphir, PhD
1
Purpose
The purpose of this workshop is to discuss why human factors engineering is important for safe and effective clinical systems and to learn the Usability methodology and how to develop and run a Usability study
3
Learning ObjectivesIdentify when to employ Usability TestingDiscern how to apply Usability testing to the
development of clinical systemsDescribe how to run a Usability studyDescribe how to interpret the results of a
usability studyDecide when not to apply Usability testingRecognize the linkage between patient safety
and usable systems
4
UsabilityUsability is the “effectiveness, efficiency and
satisfaction with which a specified set of users can achieve a specified set of tasks in a particular environment.” (International Standard Organization: ISO 9241)
Usability is the EXTENT to which a product, application, is « usable »A product is more or less usableUsability can be measured
5
Human-centred designdevelopment cycle (ISO 13407)
Plan the human-centredprocess
Understand and specifythe context of use
Specify the user and organizational requirements
Producedesign solutions
Evaluate designs againstrequirements
Meetsrequirements?
6
ISO 13407 & ISO/IEC 12207ISO/IEC-12207 primary lifecycle
Acquisit. & supply
DevelopmentOperation & mainten.Require.
anal.Architect. Design
Qualific. testing
ISO13407
Context of use N/A N/A N/A
User & org. requir. N/A N/A N/A
Design solutions N/A N/A
Evaluate
7
UsabilityEffectiveness: accuracy and completeness with which users
achieve specific goals
Efficiency: resources expended in relation with effectiveness
Satisfaction: comfort and acceptability of use
Context of use: users, goals, tasks, equipment, physical and
social environment
Task: a goal (what’s to be done) and the conditions
(constraints) to reach the goal
Activity: the set of actions (mental and behavioural) carried out
by the user to perform the task
8
Usability (cont’d)The evaluation of the usability of a product can
have different objectives:Improve the usability of an application (formative
evaluation): Usability EngineeringDuring the design and development phase of
this application (traditional usability engineering lifecycles)
Via the re-engineering of an existing applicationAcquire or not the product, choose the most usable
product (normative evaluation)
9
Usability engineering« Usability engineering provides structured
methods for achieving usability in user interface design during product development » (Deborah Mayhew, 1999)
Usability Engineering is intertwined with Software Engineering and integrated in the Project lifecycle
10
Usability engineering
Understand the project (goals, expected benefits)
Modelof the currentwork situation
Model of there-engineeredwork situationUsability goals
Iterative usability assessment
Usability problemsrecommendations
Analysis of the current work situation
• Analysis, description and model of the context of use
• Analysis, description and model of users, their tasks and activity
• Usability assessment of existing systems
Project lifecycleSystem engineering
Goals, Expected benefits (quality, safety, investment)
Users needsRequirements
SpecificationDevelopment of a concept
Implementation of theconcept
Release
Requirements verification
Specification verification
Product verificationIterative testing
Validation
11
The usability engineering toolboxTheories, conceptual frameworks, models,
methods, techniques, devices …
From various fields: cognitive psychology, cognitive ergonomics, cognitive engineering, usability, ethnology, sociology, software engineering …
12
COGNITIVE psychology, ergonomics, engineering
Theories and modelsDecision making, process control, human error
Methodologies, frameworksCognitive Task Analysis (CTA); activity analysis
MethodsInterviews, questionnaires, protocol analysis,
documents analysis, audio/video-recorded observations, …
13
UsabilityMethods:
Usability inspection: heuristic evaluation, cognitive walkthrough
Usability testsFocus groups
Techniques, devices:Usability labs
14
Rationale for the use of HFESo What?
Who Cares?
What is in it for you?
15
So What?IBM – “2/3 of all healthcare dollars spent on IT
are spent on Systems which are never used or are used for less than 3 months.”
Patient Safety can be compromised by poorly designed systemsHas led to deaths Is a source of unnecessary liability
16
Who Cares?Administration
Limiting Costs (Mayo saved 1.5M, day 1)Physicians
Limiting ErrorsIncreased Efficiency
PatientsGreater Safety
17
What is in it for you?Greater ReliabilityGreater AcceptanceLower Cost of ImplementationLower Cost for TrainingIncreased Patient SafetyImproved Clinical Reputation of your
Organization
18
Reasons for Hard-to-Use Products
19
Reasons for Hard-to-Use ProductsDevelopment Emphasis was on the Machine or
System, Rather than the User
20
Reasons for Hard-to-Use ProductsDevelopment Emphasis was on the Machine or
System, Rather than the UserTarget Audience is a Moving Target
21
Reasons for Hard-to-Use ProductsDevelopment Emphasis was on the Machine or
System, Rather than the UserTarget Audience is a Moving TargetLack of Design Verification (Science vs.
“Common Sense”)
22
Reasons for Hard-to-Use ProductsDevelopment Emphasis was on the Machine or
System, Rather than the UserTarget Audience is a Moving TargetLack of Design Verification (Science vs.
“Common Sense”)Development Teams are not well Integrated
23
Reasons for Hard-to-Use ProductsDevelopment Emphasis was on the Machine or
System, Rather than the UserTarget Audience is a Moving TargetLack of Design Verification (Science vs.
“Common Sense”)Development Teams are not well IntegratedSkills necessary for the interface Design are
Different than those necessary for the Technical Implementation of Systems!
24
User-Centered Design
25
User-Centered DesignRequires an early focus on users and tasks
26
User-Centered DesignRequires an early focus on users and tasksRequires empiric measurement of product usage
27
User-Centered DesignRequires an early focus on users and tasksRequires empiric measurement of product usageRequires iterative design cycles
DesignTestingModificationRe-Design
28
Human Factors Analysis
29
Human Factors AnalysisContextual Inquiry
Understanding End-User Needs
30
Human Factors AnalysisContextual Inquiry
Understanding End-User NeedsCompetitive Usability Evaluations
Expose Gaps in Existing Systems
31
Human Factors AnalysisContextual Inquiry
Understanding End-User NeedsCompetitive Usability Evaluations
Expose Gaps in Existing SystemsLow-Fidelity Prototyping
Rapid turnaround of Identified Program Changes
32
Human Factors AnalysisContextual Inquiry
Understanding End-User NeedsCompetitive Usability Evaluations
Expose Gaps in Existing SystemsLow-Fidelity Prototyping
Rapid turnaround of Identified Program Changes
Modified Focus GroupsUnderstand Relevant Workflow and Processes
33
Human Factors AnalysisContextual Inquiry
Understanding End-User NeedsCompetitive Usability Evaluations
Expose Gaps in Existing SystemsLow-Fidelity Prototyping
Rapid turnaround of Identified Program Changes
Modified Focus GroupsUnderstand Relevant Workflow and Processes
34
Usability Studies
35
Usability StudiesDevelopers and Evaluators
36
Usability StudiesDevelopers and EvaluatorsObserve without Intervention
37
Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users Pain
38
Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with Imperfection
39
Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the User
40
Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the UserResist the Impulse to Blame Yourself
41
Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the UserResist the Impulse to Blame YourselfAcceptance
42
Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the UserResist the Impulse to Blame YourselfAcceptanceProgress
43
Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the UserResist the Impulse to Blame YourselfAcceptanceProgress
44
Realities of Human Factors Engineering ~Jakob NielsenYour Best Guess Is Not Good Enough
It is impossible to design an optimal user interface just by giving it your best try.
Users will make unexpected misinterpretations of the interface and perform tasks differently than you expect.
45
The User is Always RightAccept the need to make modifications to fit the
users’ needs and expectations.If users have problems with the software, it is not
their fault.
46
The User is Not Always RightCaution: But it does not follow that the most
usable interfaces can be designed just by asking users what they would like. Users may not understand how changes/enhancements could be beneficial.
47
Designers Are Not UsersDesigners have a great deal of computer
experienceDesigners have an inherent enthusiasm about
computer applicationsDesigners know the conceptual foundation for
the design of the interface“Knowing about a system is a one-way street.
One cannot go back to knowing nothing.”
48
Vice Presidents Are Not Typical UsersCompany executives typically have very different
characteristics than the user population.
49
Usability Engineering is a ProcessThe Usability Engineering process is well refined
and the activities needed to arrive at a good result are fairly constant
However, each project is different and will have different objectives.
50
Why GUI Design Fails
51
Why GUI Design FailsProgrammers/Designers
52
Why GUI Design FailsProgrammers/Designers
Design to their work patterns, not user work patterns
53
Why GUI Design FailsProgrammers/Designers
Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptions
54
Why GUI Design FailsProgrammers/Designers
Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasks
55
Why GUI Design FailsProgrammers/Designers
Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”
56
Why GUI Design FailsProgrammers/Designers
Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”Plan that users will refer to documentation
57
Why GUI Design FailsProgrammers/Designers
Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”Plan that users will refer to documentationProvide all features at the top levels
58
Why GUI Design FailsProgrammers/Designers
Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”Plan that users will refer to documentationProvide all features at the top levelsAre unaware of consistency issues or standards
59
Why GUI Design FailsProgrammers/Designers
Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”Plan that users will refer to documentationProvide all features at the top levelsAre unaware of consistency issues or standards
60
Bird Flu
61
Usefulness: What Does This Mean?
Utility - does the system encompass needed functionality
Easy to learn
Efficient to use
Easy to remember
Few Errors
Subjectively pleasing
Usefulness -can system be used to achieve desired goals?
Usability -how well users can use functionality?
62
What is a Usability Study?A usability study is a replicable study of
consumer/product or consumer/process interactions conducted in a controlled, simulated environment.
63
Essential ComponentsTo accomplish a valid study, we follow a specific
protocol and have multiple participants interact with the same situations (scenarios).
It is important to observe several participants interacting with the software in order to identify trends and prioritize issues. The goal is to improve the process or product, not to train the participants.
64
What can be studied?Products or Processes True classical experiments with large sample
sizes and complex test designs Informal, less complex studies designed for
rapid processing of results
Note: Each type of study has different objectives, as well as different time and resource requirements.
65
What can be learned?
66
What can be learned?Features/Functions that work
67
What can be learned?Features/Functions that workFeatures/Functions that need improvement
68
What can be learned?Features/Functions that workFeatures/Functions that need improvementUser/Consumer work patterns and mental
models
69
What can be learned?Features/Functions that workFeatures/Functions that need improvementUser/Consumer work patterns and mental
modelsUser/Consumer expectations and desires
70
What can be learned?Features/Functions that workFeatures/Functions that need improvementUser/Consumer work patterns and mental
modelsUser/Consumer expectations and desiresPrioritize areas for improvement - unique
advantage
71
What can be learned?Features/Functions that workFeatures/Functions that need improvementUser/Consumer work patterns and mental
modelsUser/Consumer expectations and desiresPrioritize areas for improvement - unique
advantage
72
Objectives Addressed Through the Study DesignWhat are the general navigational weaknesses? Ease of use / Intuitive nature of application
interface (Do controls and functions make sense?)
User satisfaction?Efficient use of screen
real estate (space)? Are fields and widgets placed intuitively on the
screen?
73
Objectives Addressed Through the Study DesignWill the application make users’ jobs easier? /
Will it make users more productive? Is the system responsive/ fast enough?Is the complexity of the application appropriate
for the defined / intended user group(s)? What On-line Help functions are needed?What type(s) / degree of training is needed?
74
Limitations of Usability Testing
75
Limitations of Usability TestingTesting is always an artificial situation
76
Limitations of Usability TestingTesting is always an artificial situationTest Results do not Prove that a Product Works
77
Limitations of Usability TestingTesting is always an artificial situationTest Results do not Prove that a Product WorksParticipants are rarely truly representative of
their target Population
78
Limitations of Usability TestingTesting is always an artificial situationTest Results do not Prove that a Product WorksParticipants are rarely truly representative of
their target PopulationTesting is not Always the Best Technique
(To Test or Not to Test?)vs. Expert EvaluationFocus Groups
79
When Usability Study
80
When Usability StudyWhen Studying Complex Processes without
option to Study Components Separately
81
When Usability StudyWhen Studying Complex Processes without
option to Study Components SeparatelyLab Environment is Vastly different than User
Environment
82
When Usability StudyWhen Studying Complex Processes without
option to Study Components SeparatelyLab Environment is Vastly different than User
EnvironmentLab Space will not accommodate the Test
83
When Usability StudyWhen Studying Complex Processes without
option to Study Components SeparatelyLab Environment is Vastly different than User
EnvironmentLab Space will not accommodate the TestWhen feedback will not be Utilized
84
When Usability StudyWhen Studying Complex Processes without
option to Study Components SeparatelyLab Environment is Vastly different than User
EnvironmentLab Space will not accommodate the TestWhen feedback will not be UtilizedSubjects cannot be found (Typical Users)
85
When Usability StudyWhen Studying Complex Processes without
option to Study Components SeparatelyLab Environment is Vastly different than User
EnvironmentLab Space will not accommodate the TestWhen feedback will not be UtilizedSubjects cannot be found (Typical Users)
86
87
Usability Lab
88
89
90
91
92
93
94
95
96
97
98
Illustration
Overview of the re-engineering project of a CPOE application carried out at the EVALAB
Focus on usability tests of the system
99
The projectPartners:
University Hospital of LilleSyndicat Inter-hospitalier de Bretagne (SIB)
Project and context:Re-engineering of an existing Computerized
Physician Order Entry (CPOE) application: Génois
100
The project (cont’d)
Medication Ordering, [dispensing], and administration process
Hospital setting, survey of the therapeutic process during the patient’s stay
101
Medication orderingDaily medical rounds
At bedside
Follow-up of Medication
administration
Medication decision making
Medication ordering
Preparation of 24h pillboxes
Administrationvalidation
patient’s medication orders list
Usability engineering
Understand the project (goals, expected benefits)
Modelof the currentwork situation
Model of there-engineeredwork situationUsability goals
Iterative usability assessment
Usability problemsrecommendations
Analysis of the current work situation
• Analysis, description and model of the context of use
• Analysis, description and model of users, their tasks and activity
• Usability assessment of existing systems
The project: Re-engineering of an existing CPOE application: Génois (SIB)
103
Usability engineering
Understand the project (goals, expected benefits)
Modelof the currentwork situation
Model of there-engineeredwork situationUsability goals
Iterative usability assessment
Usability problemsrecommendations
Analysis of the current work situation
• Analysis, description and model of the context of use
• Analysis, description and model of users, their tasks and activity
• Usability assessment of existing systems
Goals and Expected Benefits
104
Company (SIB): Improve the usability of the
CPOE application (Génois) Support the re-engineering of
the application (re-design HCI)
Hospital: Assess the feasibility of the
installation of this CPOE Improve the quality and
safety of the medication process
Usability engineering
Understand the project (goals, expected benefits)
Modelof the currentwork situation
Model of there-engineeredwork situationUsability goals
Iterative usability assessment
Usability problemsrecommendations
Analysis of the current work situation
• Analysis, description and model of the context of use
• Analysis, description and model of users, their tasks and activity
• Usability assessment of existing systems
Identify the current work situation
105
Identify the Work SituationUsers: physicians, nursesSystems: paper-based / computer-based
106
Usability engineering
Understand the project (goals, expected benefits)
Modelof the currentwork situation
Model of there-engineeredwork situationUsability goals
Iterative usability assessment
Usability problemsrecommendations
Analysis of the current work situation
• Analysis, description and model of the context of use
• Analysis, description and model of users, their tasks and activity
• Usability assessment of existing systems
Analysis of the context of use
Analysis of Users’ tasks & activity
Usability assessment of existing systems
107
Analysis of the Work Situation (1 of 3)
Usability tasks Methods
Analysis of the context of use Interviews: head nurses, managers, senior physicianAnalysis (inventory) of systems’ functionsOn site observations (ethno methodology)
Analysis of Users’ tasks & activity•Individual
•Collective (cooperation)
Cognitive Task AnalysisActivity analysisDocuments analysis (content analysis)Self confrontation interviews
Analysis of communications (oral, written, through the system)
Usability assessment of existing systems Computer-based (CPOE):Usability inspection Usability tests
Paper-based:Analysis of the format, content, availability, redundancy of supports / contentsUsers complaints and incidents observed or reported
108
Analysis of the Work Situation (2 of 3)
Usability tasks Methods
Analysis of the context of use Interviews: head nurses, managers, senior physicianAnalysis (inventory) of systems’ functionsOn site observations (ethno methodology)
Analysis of Users’ tasks & activity•Individual
•Collective (cooperation)
Cognitive Task AnalysisActivity analysisDocuments analysis (content analysis)Self confrontation interviews
Analysis of communications (oral, written, through the system)
Usability assessment of existing systems Computer-based (CPOE):Usability inspection Usability tests
Paper-based:Analysis of the format, content, availability, redundancy of supports / contentsUsers complaints and incidents observed or reported
109
Analysis of the Work Situation (3 of 3)Usability tasks Methods
Analysis of the context of use Interviews: head nurses, managers, senior physicianAnalysis (inventory) of systems’ functionsOn site observations (ethno methodology)
Analysis of Users’ tasks & activity•Individual
•Collective (cooperation)
Cognitive Task AnalysisActivity analysisDocuments analysis (content analysis)Self confrontation interviews
Analysis of communications (oral, written, through the system)
Usability assessment of existing systems
Computer-based (CPOE):Usability inspection Usability tests
Paper-based:Analysis of the format, content, availability, redundancy of supports / contentsUsers complaints and incidents observed or reported
110
Outcomes of the analysis of the work situation: Illustration on two tasks
Physician: information gathering / patient’s current medications for therapeutic decision making
Nurse: information gathering / patient’s current medication for the preparation of medication administration (24 hours pill-box)
111
Physician’s task: comprehend the patient’s list of meds
Usability assessment
Activity analysis• Doctors need the list of current
meds on one page• Need the duration of each
order immediately available (no calculation)
• Need the recent modifications of the treatment highlighted
Context of use• Never performs this task
alone (nurse or resident present)
• Frequency of encounters with the patient varies from twice per day to once in 2 weeks
Paper-based system• The medication order sheets (list
of orders written by the physician) are not usable
• The physicians prefer to use the nurses Medication Administration Record to update their representation of current treatments
CPOE system (inspection, on site observations)
• Obligation of scrolling through the screen page to read the entire list of meds
• Display of modifications of the treatments suffers from lack of intuitivity
• Duration is not directly available (computation required)
Nurse’s task: prepare the pillboxes (24h administration)
Usability assessment
Activity analysis• Nurses need the list of current
meds on one page, sorted out by route
• Need to be informed immediately and very clearly of the medication changes ordered by the doctors along their rounds or visits
Context of use• No problem when the
nurses accompany the doctors along their rounds
• If not, great difficulties to update their knowledge about the patients current treatment and to remain aware of the modifications of orders
CPOE system (on-site observations)
• Difficulties with the 24h administration schedule
• The system requires the printing of the preparation plans: in case of frequent modifications of the treatment, there are a lot of printed plans, and a risk of confusion
Paper-based system (on-site observations)• The medication order list is not
usable• The only synthesized
document (MAR) is not available at the location where the pillboxes are prepared
Usability tests (1) PhysiciansPhysicians
Test the usability of information display for the information gathering task supporting the decision making
Check for recent changes in the current therapeutic treatment
Test orders entry functions
114
Usability tests (2) NursesNurses
Test the usability of the functions supporting the preparation of the 24h pillboxes (display of information, editing of preparation plans)
Test the usability of administration validation
115
Usability testsThe usability labPreparation of the testsExcerpts (video)ConclusionsRecommendations
116
Observers screens
Observers
Technician’scontrol
Facilitator control screens
Control roomDirect observation room
Main experimental&
simulation room
Experimental & simulation room
Experimental & simulation room
One-way mirrors
Usability testsThe usability labPreparation of the testsExcerpts (video)ConclusionsRecommendations
118
Create a realistic simulated environmentConfigure the system to accommodate the
characteristics of users activityCreate realistic patient’s records (from real
cases)Write precise scenarios
119
RecommendationsItems in the Interface that worked well and safelyItems in the Interface that worked poorly or
inconsistentlyIdentify those features that are capable of causing
Harm.Recommendations for specific engineering
improvements to the interface design
120
Conclusions
121
ConclusionsUsability Studies
122
ConclusionsUsability Studies
Useful Mechanisms for Objective Evaluation
123
ConclusionsUsability Studies
Useful Mechanisms for Objective EvaluationDesigned to Answer Specific Questions
124
ConclusionsUsability Studies
Useful Mechanisms for Objective EvaluationDesigned to Answer Specific QuestionsDesigned to Discover Questions to Answer
125
ConclusionsUsability Studies
Useful Mechanisms for Objective EvaluationDesigned to Answer Specific QuestionsDesigned to Discover Questions to AnswerKeeping patients safe from software related medical
error
126
“The Best Way to Predict the Future,
127
“The Best Way to Predict the Future,
Is to Create It.”
128
Quote: Peter Drucker – Harvard University
“The Best Way to Predict the Future,
Is to Create It.”
129