WCIT 2014 Peter Elkin - Human computer interaction, evaluation, usability testing, study design and...

Human Computer Interaction Evaluation, usability testing,

study design and methods

Presented by: Peter L. Elkin, MD, MACP, FACMI

With Studies from EvaLAB

Marie-Catherine Beuscart-Zéphir, PhD

Purpose

The purpose of this workshop is to discuss why human factors engineering is important for safe and effective clinical systems and to learn the Usability methodology and how to develop and run a Usability study

Learning ObjectivesIdentify when to employ Usability TestingDiscern how to apply Usability testing to the

development of clinical systemsDescribe how to run a Usability studyDescribe how to interpret the results of a

usability studyDecide when not to apply Usability testingRecognize the linkage between patient safety

and usable systems

UsabilityUsability is the “effectiveness, efficiency and

satisfaction with which a specified set of users can achieve a specified set of tasks in a particular environment.” (International Standard Organization: ISO 9241)

Usability is the EXTENT to which a product, application, is « usable »A product is more or less usableUsability can be measured

Human-centred designdevelopment cycle (ISO 13407)

Plan the human-centredprocess

Understand and specifythe context of use

Specify the user and organizational requirements

Producedesign solutions

Evaluate designs againstrequirements

Meetsrequirements?

ISO 13407 & ISO/IEC 12207ISO/IEC-12207 primary lifecycle

Acquisit. & supply

DevelopmentOperation & mainten.Require.

anal.Architect. Design

Qualific. testing

ISO13407

Context of use N/A N/A N/A

User & org. requir. N/A N/A N/A

Design solutions N/A N/A

Evaluate

UsabilityEffectiveness: accuracy and completeness with which users

achieve specific goals

Efficiency: resources expended in relation with effectiveness

Satisfaction: comfort and acceptability of use

Context of use: users, goals, tasks, equipment, physical and

social environment

Task: a goal (what’s to be done) and the conditions

(constraints) to reach the goal

Activity: the set of actions (mental and behavioural) carried out

by the user to perform the task

Usability (cont’d)The evaluation of the usability of a product can

have different objectives:Improve the usability of an application (formative

evaluation): Usability EngineeringDuring the design and development phase of

this application (traditional usability engineering lifecycles)

Via the re-engineering of an existing applicationAcquire or not the product, choose the most usable

product (normative evaluation)

Usability engineering« Usability engineering provides structured

methods for achieving usability in user interface design during product development » (Deborah Mayhew, 1999)

Usability Engineering is intertwined with Software Engineering and integrated in the Project lifecycle

Usability engineering

Understand the project (goals, expected benefits)

Modelof the currentwork situation

Model of there-engineeredwork situationUsability goals

Iterative usability assessment

Usability problemsrecommendations

Analysis of the current work situation

• Analysis, description and model of the context of use

• Analysis, description and model of users, their tasks and activity

• Usability assessment of existing systems

Project lifecycleSystem engineering

Goals, Expected benefits (quality, safety, investment)

Users needsRequirements

SpecificationDevelopment of a concept

Implementation of theconcept

Release

Requirements verification

Specification verification

Product verificationIterative testing

Validation

The usability engineering toolboxTheories, conceptual frameworks, models,

methods, techniques, devices …

From various fields: cognitive psychology, cognitive ergonomics, cognitive engineering, usability, ethnology, sociology, software engineering …

COGNITIVE psychology, ergonomics, engineering

Theories and modelsDecision making, process control, human error

Methodologies, frameworksCognitive Task Analysis (CTA); activity analysis

MethodsInterviews, questionnaires, protocol analysis,

documents analysis, audio/video-recorded observations, …

UsabilityMethods:

Usability inspection: heuristic evaluation, cognitive walkthrough

Usability testsFocus groups

Techniques, devices:Usability labs

Rationale for the use of HFESo What?

Who Cares?

What is in it for you?

So What?IBM – “2/3 of all healthcare dollars spent on IT

are spent on Systems which are never used or are used for less than 3 months.”

Patient Safety can be compromised by poorly designed systemsHas led to deaths Is a source of unnecessary liability

Who Cares?Administration

Limiting Costs (Mayo saved 1.5M, day 1)Physicians

Limiting ErrorsIncreased Efficiency

PatientsGreater Safety

What is in it for you?Greater ReliabilityGreater AcceptanceLower Cost of ImplementationLower Cost for TrainingIncreased Patient SafetyImproved Clinical Reputation of your

Organization

Reasons for Hard-to-Use Products

Reasons for Hard-to-Use ProductsDevelopment Emphasis was on the Machine or

System, Rather than the User

System, Rather than the UserTarget Audience is a Moving Target

System, Rather than the UserTarget Audience is a Moving TargetLack of Design Verification (Science vs.

“Common Sense”)

“Common Sense”)Development Teams are not well Integrated

“Common Sense”)Development Teams are not well IntegratedSkills necessary for the interface Design are

Different than those necessary for the Technical Implementation of Systems!

User-Centered Design

User-Centered DesignRequires an early focus on users and tasks

User-Centered DesignRequires an early focus on users and tasksRequires empiric measurement of product usage

User-Centered DesignRequires an early focus on users and tasksRequires empiric measurement of product usageRequires iterative design cycles

DesignTestingModificationRe-Design

Human Factors Analysis

Human Factors AnalysisContextual Inquiry

Understanding End-User Needs

Understanding End-User NeedsCompetitive Usability Evaluations

Expose Gaps in Existing Systems

Expose Gaps in Existing SystemsLow-Fidelity Prototyping

Rapid turnaround of Identified Program Changes

Modified Focus GroupsUnderstand Relevant Workflow and Processes

Usability Studies

Usability StudiesDevelopers and Evaluators

Usability StudiesDevelopers and EvaluatorsObserve without Intervention

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users Pain

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with Imperfection

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the User

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the UserResist the Impulse to Blame Yourself

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the UserResist the Impulse to Blame YourselfAcceptance

Usability StudiesDevelopers and EvaluatorsObserve without InterventionFeel the Users PainCome to Grips with ImperfectionResist the Impulse to Blame the UserResist the Impulse to Blame YourselfAcceptanceProgress

Realities of Human Factors Engineering ~Jakob NielsenYour Best Guess Is Not Good Enough

It is impossible to design an optimal user interface just by giving it your best try.

Users will make unexpected misinterpretations of the interface and perform tasks differently than you expect.

The User is Always RightAccept the need to make modifications to fit the

users’ needs and expectations.If users have problems with the software, it is not

their fault.

The User is Not Always RightCaution: But it does not follow that the most

usable interfaces can be designed just by asking users what they would like. Users may not understand how changes/enhancements could be beneficial.

Designers Are Not UsersDesigners have a great deal of computer

experienceDesigners have an inherent enthusiasm about

computer applicationsDesigners know the conceptual foundation for

the design of the interface“Knowing about a system is a one-way street.

One cannot go back to knowing nothing.”

Vice Presidents Are Not Typical UsersCompany executives typically have very different

characteristics than the user population.

Usability Engineering is a ProcessThe Usability Engineering process is well refined

and the activities needed to arrive at a good result are fairly constant

However, each project is different and will have different objectives.

Why GUI Design Fails

Why GUI Design FailsProgrammers/Designers

Design to their work patterns, not user work patterns

Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptions

Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasks

Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”

Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”Plan that users will refer to documentation

Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”Plan that users will refer to documentationProvide all features at the top levels

Design to their work patterns, not user work patternsDesign to their metaphor, not user perceptionsWant the application to control user tasksAssume all users will be “Trained”Plan that users will refer to documentationProvide all features at the top levelsAre unaware of consistency issues or standards

Bird Flu

Usefulness: What Does This Mean?

Utility - does the system encompass needed functionality

Easy to learn

Efficient to use

Easy to remember

Few Errors

Subjectively pleasing

Usefulness -can system be used to achieve desired goals?

Usability -how well users can use functionality?

What is a Usability Study?A usability study is a replicable study of

consumer/product or consumer/process interactions conducted in a controlled, simulated environment.

Essential ComponentsTo accomplish a valid study, we follow a specific

protocol and have multiple participants interact with the same situations (scenarios).

It is important to observe several participants interacting with the software in order to identify trends and prioritize issues. The goal is to improve the process or product, not to train the participants.

What can be studied?Products or Processes True classical experiments with large sample

sizes and complex test designs Informal, less complex studies designed for

rapid processing of results

Note: Each type of study has different objectives, as well as different time and resource requirements.

What can be learned?

What can be learned?Features/Functions that work

What can be learned?Features/Functions that workFeatures/Functions that need improvement

What can be learned?Features/Functions that workFeatures/Functions that need improvementUser/Consumer work patterns and mental

models

modelsUser/Consumer expectations and desires

modelsUser/Consumer expectations and desiresPrioritize areas for improvement - unique

advantage

modelsUser/Consumer expectations and desiresPrioritize areas for improvement - unique

advantage

Objectives Addressed Through the Study DesignWhat are the general navigational weaknesses? Ease of use / Intuitive nature of application

interface (Do controls and functions make sense?)

User satisfaction?Efficient use of screen

real estate (space)? Are fields and widgets placed intuitively on the

screen?

Objectives Addressed Through the Study DesignWill the application make users’ jobs easier? /

Will it make users more productive? Is the system responsive/ fast enough?Is the complexity of the application appropriate

for the defined / intended user group(s)? What On-line Help functions are needed?What type(s) / degree of training is needed?

Limitations of Usability Testing

Limitations of Usability TestingTesting is always an artificial situation

Limitations of Usability TestingTesting is always an artificial situationTest Results do not Prove that a Product Works

Limitations of Usability TestingTesting is always an artificial situationTest Results do not Prove that a Product WorksParticipants are rarely truly representative of

their target Population

Limitations of Usability TestingTesting is always an artificial situationTest Results do not Prove that a Product WorksParticipants are rarely truly representative of

their target PopulationTesting is not Always the Best Technique

(To Test or Not to Test?)vs. Expert EvaluationFocus Groups

When Usability Study

When Usability StudyWhen Studying Complex Processes without

option to Study Components Separately

option to Study Components SeparatelyLab Environment is Vastly different than User

Environment

EnvironmentLab Space will not accommodate the Test

EnvironmentLab Space will not accommodate the TestWhen feedback will not be Utilized

EnvironmentLab Space will not accommodate the TestWhen feedback will not be UtilizedSubjects cannot be found (Typical Users)

Usability Lab

Illustration

Overview of the re-engineering project of a CPOE application carried out at the EVALAB

Focus on usability tests of the system

The projectPartners:

University Hospital of LilleSyndicat Inter-hospitalier de Bretagne (SIB)

Project and context:Re-engineering of an existing Computerized

Physician Order Entry (CPOE) application: Génois

The project (cont’d)

Medication Ordering, [dispensing], and administration process

Hospital setting, survey of the therapeutic process during the patient’s stay

Medication orderingDaily medical rounds

At bedside

Follow-up of Medication

administration

Medication decision making

Medication ordering

Preparation of 24h pillboxes

Administrationvalidation

patient’s medication orders list

The project: Re-engineering of an existing CPOE application: Génois (SIB)

Goals and Expected Benefits

Company (SIB): Improve the usability of the

CPOE application (Génois) Support the re-engineering of

the application (re-design HCI)

Hospital: Assess the feasibility of the

installation of this CPOE Improve the quality and

safety of the medication process

Identify the current work situation

Identify the Work SituationUsers: physicians, nursesSystems: paper-based / computer-based

Analysis of the context of use

Analysis of Users’ tasks & activity

Usability assessment of existing systems

Analysis of the Work Situation (1 of 3)

Usability tasks Methods

Analysis of the context of use Interviews: head nurses, managers, senior physicianAnalysis (inventory) of systems’ functionsOn site observations (ethno methodology)

Analysis of Users’ tasks & activity•Individual

•Collective (cooperation)

Cognitive Task AnalysisActivity analysisDocuments analysis (content analysis)Self confrontation interviews

Analysis of communications (oral, written, through the system)

Usability assessment of existing systems Computer-based (CPOE):Usability inspection Usability tests

Paper-based:Analysis of the format, content, availability, redundancy of supports / contentsUsers complaints and incidents observed or reported

Analysis of the Work Situation (2 of 3)

Usability tasks Methods

Usability assessment of existing systems Computer-based (CPOE):Usability inspection Usability tests

Analysis of the Work Situation (3 of 3)Usability tasks Methods

Usability assessment of existing systems

Computer-based (CPOE):Usability inspection Usability tests

Outcomes of the analysis of the work situation: Illustration on two tasks

Physician: information gathering / patient’s current medications for therapeutic decision making

Nurse: information gathering / patient’s current medication for the preparation of medication administration (24 hours pill-box)

Physician’s task: comprehend the patient’s list of meds

Usability assessment

Activity analysis• Doctors need the list of current

meds on one page• Need the duration of each

order immediately available (no calculation)

• Need the recent modifications of the treatment highlighted

Context of use• Never performs this task

alone (nurse or resident present)

• Frequency of encounters with the patient varies from twice per day to once in 2 weeks

Paper-based system• The medication order sheets (list

of orders written by the physician) are not usable

• The physicians prefer to use the nurses Medication Administration Record to update their representation of current treatments

CPOE system (inspection, on site observations)

• Obligation of scrolling through the screen page to read the entire list of meds

• Display of modifications of the treatments suffers from lack of intuitivity

• Duration is not directly available (computation required)

Nurse’s task: prepare the pillboxes (24h administration)

Usability assessment

Activity analysis• Nurses need the list of current

meds on one page, sorted out by route

• Need to be informed immediately and very clearly of the medication changes ordered by the doctors along their rounds or visits

Context of use• No problem when the

nurses accompany the doctors along their rounds

• If not, great difficulties to update their knowledge about the patients current treatment and to remain aware of the modifications of orders

CPOE system (on-site observations)

• Difficulties with the 24h administration schedule

• The system requires the printing of the preparation plans: in case of frequent modifications of the treatment, there are a lot of printed plans, and a risk of confusion

Paper-based system (on-site observations)• The medication order list is not

usable• The only synthesized

document (MAR) is not available at the location where the pillboxes are prepared

Usability tests (1) PhysiciansPhysicians

Test the usability of information display for the information gathering task supporting the decision making

Check for recent changes in the current therapeutic treatment

Test orders entry functions

Usability tests (2) NursesNurses

Test the usability of the functions supporting the preparation of the 24h pillboxes (display of information, editing of preparation plans)

Test the usability of administration validation

Usability testsThe usability labPreparation of the testsExcerpts (video)ConclusionsRecommendations

Observers screens

Observers

Technician’scontrol

Facilitator control screens

Control roomDirect observation room

Main experimental&

simulation room

Experimental & simulation room

One-way mirrors

Usability testsThe usability labPreparation of the testsExcerpts (video)ConclusionsRecommendations

Create a realistic simulated environmentConfigure the system to accommodate the

characteristics of users activityCreate realistic patient’s records (from real

cases)Write precise scenarios

RecommendationsItems in the Interface that worked well and safelyItems in the Interface that worked poorly or

inconsistentlyIdentify those features that are capable of causing

Harm.Recommendations for specific engineering

improvements to the interface design

Conclusions

ConclusionsUsability Studies

Useful Mechanisms for Objective Evaluation

Useful Mechanisms for Objective EvaluationDesigned to Answer Specific Questions

Useful Mechanisms for Objective EvaluationDesigned to Answer Specific QuestionsDesigned to Discover Questions to Answer

Useful Mechanisms for Objective EvaluationDesigned to Answer Specific QuestionsDesigned to Discover Questions to AnswerKeeping patients safe from software related medical

“The Best Way to Predict the Future,

Is to Create It.”

Quote: Peter Drucker – Harvard University

“The Best Way to Predict the Future,

Is to Create It.”

WCIT 2014 Peter Elkin - Human computer interaction, evaluation, usability testing, study design and...

Healthcare

Apresentação WCIT 2016

WCIT - 12 Dubai 2012

WCIT 2016 Aline Sordilli

David Elkin (2)

Chapayesica Elkin

WCIT 2016 Soumitra Dutta

Documento de cierre de la WCIT

Manuel Elkin

Day1 WCIT 2014

Elkin antonio

WCIT-12 FINAL ACTS – ITU

WCIT-12 Final Acts (Dubai 2012)

What happened at wcit in december 2012

WCIT 2010 - a New City

WCIT 2014 Gary Beach - The Skills Gap

WCIT Postmortem: The Lessons Learned · WCIT Postmortem: The Lessons Learned Sally Wentworth Senior Manager, Public Policy

Elkin present

Aporte Elkin Padilla

WCIT Sally Wentworth Internet Society 1. What we will cover Context – setting the stage WCIT – background and preparations WCIT what happened 2012 ITRs

Elkin alejandro