Embed Size (px)
Citation preview
Engineering the Future of Healthcare: Harnessing Health IT, Design, and Systems SafetyKristen Miller, DrPH, CPPSScientific Director, National Center for Human Factors in HealthcareAssociate Professor of Emergency Medicine, Georgetown University
@KMillerDrPH@MedicalHFE
Training:• DrPH Human Factors Engineering and Ergonomics• Armstrong Institute for Patient Safety and Quality• Patient Safety Fellowship, Dept. Veteran’s Affairs• Director of Human Factors, Christiana Care Value Institute• Scientific Director, National Center for Human Factors
Affiliations: • Associate Professor, Georgetown University School of Medicine• Affiliate Faculty, Innovation Center for Biomedical Informatics• Adjunct Faculty, Catholic University Biomedical Engineering• Faculty, ACM Computer-Human Interaction Summer School in e-Health and m-Health• Senior Advisor, Dental Patient Safety Foundation
Introduction
• Identify the need for system solutions to support the way humans work, minimize opportunity for error, and empower teams to delivery high quality care.
• Apply human factors engineering concepts to the design, development, and deployment of health IT initiatives.
• Consider how the advances in healthcare technology are influencing healthcare delivery and how to improve health IT usability.
Learning Objectives
Agenda
• Overview: National Center for Human Factors in Healthcare
• Human Factors 101: Designing Safety into Healthcare Systems
• Optimizing CDS
– Right Data– Right Design– Usability– Comprehensive Evaluation
OVERVIEW
MedStar HealthNational Center for Human Factors in Healthcare
We focus on studying human capabilities and designing technology, systems, and processes to meet these capabilities for safety, efficiency, & quality.
30+ Team Members• Human Factors• Health Equity• Computer Science• Aerospace Engineering• Clinicians• Environmental Design• Usability• Safety
National Center for Human Factors in Healthcare
Applied Research• Grants and contracts from government, foundations, and industry• Publications, presentations, interventions, policy recommendations
Usability Services• Medical devices• Digital health
Safety Integration• Safety consults• Serious safety event reviews
Education and Outreach• Georgetown University School of Medicine• Workshops, talks, and trainings
The Center’s Work
“We don’t redesign humans; We redesign the system within which humans work”
The Central Tenant of Human Factors
The discovery and application of information about human behavior, abilities, and other characteristics to the design of tools, machines, systems, tasks, jobs, and environments for productive,
safe, comfortable, and effective human use.
Emphasis on user- or person- centered design
What is Human Factors?
• Using methods to gather unique information on:
– Hidden needs of the end-user
– Unexpected interactions between the system and the end-user
• Creating deliberate design to promote safe, efficient, effective, and timely clinical care by:
– Making it easier to do the right thing
– Making it harder to do the wrong thing
What is Human Factors?
HUMAN FACTORS 101: DESIGNING SAFETY INTO HEALTHCARE SYSTEMS
How many of you have…• Purchased the wrong grocery item because the labels looked similar?
Why Human Factors?
How many of you have…• Washed your hair with conditioner instead of shampoo?
Why Human Factors?
Why Human Factors?
How many of you have…• Walked into the wrong restroom?
Why Human Factors?
Why Human Factors?
Why Human Factors?
How many of you have…• Ever forgotten to attach something to an email?
Why Human Factors?
Human Factors Applied in Industry
Human Factors Applied in Industry
Human Factors Applied in Industry
Human Factors Applied in Industry
Human Factors Applied in Industry
Human Factors Applied in Industry
Human Factors in Healthcare
What would happen if you confused the following products?
Human Factors in Healthcare
• People tend to blame others (or themselves) when things go wrong. Hence the term “human error”.
• Often, its bad design that leads people to make errors.
Human Error
• Under normal conditions, humans have 5-7 errors per hour.
• Under stressful/ emergency/ unusual conditions, humans have an average of 11-15 errors per hour.
• Performance is made worse when the person is:
– In a hurry
– Under a high workload
– Doing more than one thing at a time
– Doing the same thing over and over
Humans are fallible.
Can you count the “F”s in the following sentence?
FINISHED FILES ARE THE RESULT OF YEARS OF SCIENTIFIC STUDY
COMBINED WITH THE EXPERIENCE OF YEARS.
For some obscure reasons, our brains do not count the “f” in “of”, maybe because the phonetic is similar to “ov”, or because during quick reading, the brain focuses on “lexical” words, and not so much on “grammatical words”.
Double checking is a standard practice in many areas of healthcare, notwithstanding the lack of evidence supporting its efficacy.
• If one human makes a mistake, another is likely to make the same mistake, missing the error
• Diffusion of responsibility
• Confirmation bias
• Deference to authority
Inspection is not reliable.
Aoccdrnig to rscheearch at an Elingsh uinervtisy, itdeosn't mttaer in waht oredr the ltteers in a wrod are,olny taht the frist and lsat ltteres are at the rghit pcleas.The rset can be a toatl mses and you can sitll raed itwouthit a porbelm. Tihs is bcuseae we do not raed erveylteter by ilstef, but the wrod as a wlohe.
Please be as careful as possible as you read this!
Palese be as cerfaul as pisobsle as you raed tihs!
• Grab a partner.
The MYTH of Multitasking
• Performing two tasks in parallel– 9.2 multi-tasking events per hour in an emergency department– 17.4 per hour on hospital wards
• Impacts:– Increased time to task completion– Increased stress– Possible memory lapses– Subsequent errors and accidents
• Health information technology further increases the opportunities
The MYTH of Multitasking
• Remember that list of words I asked you to remember?• Take a blank sheet of paper and write down as many of
them as you can.
• (No helping each other, please)
Memory
• Was the word “snooze” on the list?• How about “apple”?
• Nap?• Earthquake?• Dream?
• Sleep?
Memory
Memory
Bed, rest, awake, tired, dream, wake, snooze, earthquake,
blanket, doze, slumber, snore, nap, peace, yawn, drowsy
• Limited attentional resources: all individuals experience severe limitations in how much mental activity they can engage in due to limited cognitive resources.
• The brain has limited attentional resources.– We can concentrate on, at most, 2-3 things simultaneously.
• The brain has a limited working memory.– Most people can reliability remember 5-7 items at a time.
The Root of Human Error… The Brain
• ½ subjects were given a two-digit number to remember.• ½ subjects were given a seven-digit number to remember.
• They were sent down the hall to another room to complete the study but along the way…
Interesting Side Effects…
HINT: “We don’t redesign humans; We redesign the system within which humans work”
So what do we do to mitigate the harm of human fallibility?
HARNESSING HEALTH IT, DESIGN, AND SYSTEMS SAFETY
The Intersection of Patients, Providers, and Health Information Technology
Patient-provider communication is important!• Improved satisfaction• Improved compliance• Improved decision making• Better health outcomes• Decreased malpractice claims
The Reality…
Optimizing Health IT as a Tool within the Patient Provider Interaction• Essential for diagnosing and treating illness• Essential in establishing a meaningful patient-provider relationship• Facilitates education and counseling patients
Right Data Right Design
Advancing Clinical Knowledge at the Point of Care
Usability
Comprehensive Evaluation
EHR Perceived Trajectory
CDS is defined as “providing clinicians with clinical knowledge and patient-related
information, intelligently filtered, and presented at appropriate times to enhance patient care.”
LaRosa JA, Ahmad N, Feinberg M, Shah M, DiBrienza R, and Studer S. The use of an early alert system to improve compliance with sepsis bundles and to assess impact on mortality. Crit Care Res
Practice. 2012;2012:980369.
Clinical Decision Support
THE
Osheroff JA: Improving Medication Use and Outcomes with Clinical Decision Support: A Step-by-Step Guide. 2009, Chicago, IL: Health Information and Management Systems Society
5 RIGHTSof Clinical Decision Support
Provision of the right (correct) information
to the right person
to the right personin the right format
through the right channel
at the right time in the workflow
THE
Osheroff JA: Improving Medication Use and Outcomes with Clinical Decision Support: A Step-by-Step Guide. 2009, Chicago, IL: Health Information and Management Systems Society
5 RIGHTSof Clinical Decision Support
Provision of the right (correct) information
to the right person
to the right personin the right format
through the right channel
at the right time in the workflow
“Right Data”
Signal Detection Theory
noise signal
Signal Detection Theorycriterion
False AlarmMissed Alarm
noise signal
Setting the Thresholds
Setting the Thresholds
Alert Fatigue
Mental state that is the result of alerts consuming too much time and mental energy which can
cause relevant alerts to be unjustifiably overridden along with clinically irrelevant ones
• Ignoring alerts• Misinterpretation of alerts• Wrong selection of handling options
Case Study #1 Spinal Cord Compression
Acute spinal cord compression (SCC) is a triple threat:• Rare• Difficult to diagnose• Likely to result in devastating, irreversible
neurological outcomes if not treated in a timely manner
The MedStar SCC Task Force convened for 3 goals:• No delay diagnosis• No delay imaging• No delay definitive care
CASE
STU
DY
#1
Machine Learning Exploration• Data: 27, 440 ED encounters [8/22/16 – 9/27/18] with SCC alerts• 395 out of 27,440 (1.4%) had a MRI_STAT ordered.• 392 features per ED encounter.
• Assumption: MRI_STAT orders indicate a high perceived likelihood of SCC by the EM physician.
• Model Inputs: SCC alert and chief complaint (free-text).
• Analysis: Random forest and logistic regression used to predict if MRI_STAT would be ordered. 10 cross-fold validation applied. Cost matrix used to reduce the number of false negatives. Cost variable was applied to each modeling approach (account for alert burden).
CASE
STU
DY
#1
Signal Detection Theory
Cord Compression -Cord Compression +
SCC Detection
Tool +
SCC Detection
Tool-
Hit
True Positive (TP)
False Alarm
False Positive (FP)
Miss
False Negative (FN)
Correct Reject
True Negative (TN)
CASE
STU
DY
#1
Machine Learning ExplorationCan we predict when a MRI_STAT will be ordered?
Model classified as requiring MRI_STAT
Model classified as not requiring MRI_STAT
Requiring MRI_STAT TP FNNot requiring MRI_STAT FP TN
CASE
STU
DY
#1
Machine Learning ResultsRandom Forest
Logistic Regression
Cost 1 10 100 10^3 10^4 2x10^4 5x10^4TP 28 28 29 47 338 387 395FP 0 0 1 194 15,937 23512 26971FN 367 367 366 348 57 8 0AUC .741 .741 .757 .758 .733 .721 .703
Cost 1 10 100 10^3 10^4TP 18 103 222 270 395FP 50 893 5316 8962 27044FN 377 292 173 125 0AUC .762 .761 .749 .735 .5
CASE
STU
DY
#1
TAKEAWAY: What is in the Code?
1. Evaluate technical components• Inclusions and exclusions• Data thresholds• Data pull (e.g. fever)
2. Evaluate performance of code• Could you recreate it?• Compare results• Identify discrepancies
THE
Osheroff JA: Improving Medication Use and Outcomes with Clinical Decision Support: A Step-by-Step Guide. 2009, Chicago, IL: Health Information and Management Systems Society
5 RIGHTSof Clinical Decision Support
Provision of the right (correct) information
to the right person
to the right personin the right format
through the right channel
at the right time in the workflow
“Right Design”
Clinical Decision Support (CDS)• Effective presentation of an alert, including how and what is displayed,
may offer better cognitive support during busy patient encounters and help providers extract information quickly.
• There is little consensus on how alerts should be generated and displayed.
• Medication errors are the most common type of error in healthcare
• On average, there is at least one medication error per hospital patient per day
• Medication errors harm at least 1.5 million people per year
• Approximately $20 billion lost annually from medication errors
• Currently 50%-90% of drug safety warnings are over-ridden• Most common reasons include incorrect alert context or inappropriate
presentation of alert within context of prescribing
Case Study #2 Medication Alerts
hospital
CASE
STU
DY
#2
Background: Human Factors Principles• Information architecture and graphic interface design demand careful
consideration• Incorporating human factors principles into alerts can:
– Improve usability– Reduce workload for prescribers– Reduce prescribing errors
• User behavior and decision-making can be significantly affected by design details like:
– Navigation from one window to the next– Information placement– Font size – Information similarity– Perceived credibility
CASE
STU
DY
#2
Human Factors Approach to Alert DesignHuman Factors Principle Summary of Principle
Alarm Philosophy Logic used to clarify alert priority levels, catalogued and available. Should capture user acknowledgement and response
False Alarms Up to date, accurate, and intelligently calibrated alarms to reduce irrelevant alarm triggers
Placement Optimized alert visibility through deliberate screen placement (close proximity to current task)
Visibility Overall screen-size of the alert (target size), luminance, background contrast, and lettering characteristics.
Prioritization Match appearance of the warning to the level of hazard associated with the clinical implications of the alert. To address color-blind users’ needs, signal words and shapes can be used to communicate the priority and level of hazard
Color Used to indicate severity, type of alert, or required response. Using more than 10 colors could make it difficult for users to remember what each color indicates.
Learnability/Confusability User ability to learn and distinguish between different types of visual alerts. Fewer shared features make alerts appearmore distinct, making it easier for users to recognize and remember different alert types.
Textual Information Effective text content should contain: (1) signal word to indicate alert severity , (2) statement of the nature of the hazard, (3) instruction statement with recommended actions, (4) a consequence statement indicating the potential patient harm. Numbers (2) and (3) are the most important components.
Habituation Habituation predicts that repeated exposure to an alert that does not require that a meaningful response will result in a decrease, and eventual elimination, of responding to the alert. Draws on principles of alarm philosophy, false alarm rate, and visual distinction
Mental Model Represents the understanding individuals have about a particular topic. Given that mental models govern users’ behavior, alerting systems should adequately support pervading mental models
Proximity of Displayed Task Components
Tools for decision-making should be integrated into the body of the alert or found within close temporal and spatial proximity to the alert.
I-MeDeSA Instrument (Zachariah et al)• Instrument for evaluating the Human-Factor Principles in Medication-
Related Decision Support Alerts (I-MeDeSA)– Developed and validated to allow EHR designers to examine the compliance of
alerts with human factors principles
• I-MeDeSA scores alerts on the following nine human factors principles:– Alarm philosophy, placement, visibility, prioritization, color learnability and
confusability, text-based information, proximity of task components being displayed, and corrective actions.
• Each principle exists as a construct of individual questions that are scored.
– There are a total of 26 questions (or items) across nine constructs. Each item receives a score of ‘1’ if the item characteristic is present and a score of ‘0’ if it is absent.
CASE
STU
DY
#2
Example: Good Design
CASE
STU
DY
#2
The Special Population Warning scored the
highest (19/26)
Example: Poor Design
CASE
STU
DY
#2
The Pregnancy/Lactation
Alert scored the lowest (8/26)
I-MeDeSA Tool Validation• Regression analysis is a statistical process for estimating variable
relationships. • For alerts, as the I-MeDeSA score goes up, the number of overrides
goes down, indicating a negative correlation and implying that the tool accurately scores alerts based on the actions of providers. CA
SE S
TUD
Y #2
CDS Redesign
Why/ Risk/ Action Framework• Why the alert was triggered• The risk to the patient• Recommended actions• Uses the signal word Warning • Allows corrective action other
than acknowledgement of having seen the alert
• Color and icon (alarm philosophy)• “Opt-out” defaults
CASE
STU
DY
#2
Alternative Recommendations• Uses personalized provider information to convey relevance,
consequence, and visibility of adverse events.
CASE
STU
DY
#2
Alternative Recommendations• Increase visibility of adverse events.
CASE
STU
DY
#2
Alternative Recommendations• Increase visibility of adverse events at a local level.
CASE
STU
DY
#2
Alternative Recommendations• Increase visibility of events at a local level with provider accountability.
CASE
STU
DY
#2
Alternative Recommendations• Connect to hospital hierarchy.
CASE
STU
DY
#2
TAKEAWAY: Is the CDS visually appealing?
1. Does the design meet basic human factors principles?
2. Does the design encourage the preferred action?
THE
Osheroff JA: Improving Medication Use and Outcomes with Clinical Decision Support: A Step-by-Step Guide. 2009, Chicago, IL: Health Information and Management Systems Society
5 RIGHTSof Clinical Decision Support
Provision of the right (correct) information
to the right person
to the right personin the right format
through the right channel
at the right time in the workflow
“Usability”
• Cardiovascular disease remains the leading cause of death in the US.• The AHA/ACC recommend use of the Atherosclerotic Cardiovascular
Disease (ASCVD) risk estimator: evaluates 10-year and lifetime risk for ASCVD.
• Variables include: • Age and Race• Cholesterol levels (HDL, LDL)• Blood pressure• Use of statin therapy• Antihypertensive medication• Use of aspirin therapy• Smoking status• Diabetes status
Case Study #3 Cardiac Risk
Workflow Analysis: Methods
• Stakeholder Interviews– 6 Cardiologists, 7 Primary Care Physicians, 4 Care Navigators
• Clinical observations– 30 hours = 34 observed patient visits
• Data Analysis– “Work-as-imagined” versus “Work-as-done”
CASE
STU
DY
#3
Stakeholder Interviews: Results
3 Main Uses• To educate patients about managing cardiovascular risk.• To aide in clinical decision making about whether or not to prescribe a
statin.• To identify, in borderline cases, whether or not a patient is at risk of
cardiovascular disease.
CASE
STU
DY
#3
Clinical Observations: Goals
• Develop a detailed understanding of current physician workflow in both primary care and cardiology settings.
• Develop a detailed understanding of how and when ASCVD risk factors are addressed over the course of a typical patient exam.
• Develop detailed process maps for how ASCVD risk calculators are utilized.
CASE
STU
DY
#3
Sim
ple
Car
diol
ogy
Visi
t Pr
oces
s M
ap
CASE
STU
DY
#3
Com
plex
Prim
ary
Car
e Vi
sit
Proc
ess
Map
CASE
STU
DY
#3
Use
of A
SCVD
Ris
k C
alcu
lato
r D
urin
g Vi
sit P
roce
ss M
ap
CASE
STU
DY
#3
Functions to Features (sample)
CASE
STU
DY
#3
Function Feature Technical Requirements
Calculate ASCVD risk (with minimal clinical burden)
Autopopulate values into the ASCVD risk calculator
FHIR and CCL call
Ensure ASCVD risk is complete and up-to-date
Alert physician of missing/outdated information
FHIR and UX
Recalculate score Refresh ASCVD risk when any new calculator related value is entered
CDS hooks
Evaluate risk score trends Display risk scores (current and previously saved)
Mpage with custom FHIR component
Alert physician to change in risk score
Trigger new calculation when patient undergoes outside procedure (e.g., surgery, ED) that changes risk variables given set parameter (e.g., change in risk level)
Message center (MedConnectcore functionality)
FHIR
Cerner SmartZone
Risk Estimator (Clinician Facing)
CASE
STU
DY
#3
Risk Educator (Patient Facing)
CASE
STU
DY
#3
Usability Testing: Methods
• Formative Usability Testing– 8 Cardiologists, 7 Primary
Care Physicians
• Areas of Interest for our Prototype– Display– General Content– Functionality
CASE
STU
DY
#3
TAKEAWAY: How will the tool be implemented?
1. Are true end-users involved?2. Have we assessed effectiveness? Efficiency? Satisfaction?3. Does current implementation support sustainability?
THE
Osheroff JA: Improving Medication Use and Outcomes with Clinical Decision Support: A Step-by-Step Guide. 2009, Chicago, IL: Health Information and Management Systems Society
5 RIGHTSof Clinical Decision Support
Provision of the right (correct) information
to the right person
to the right personin the right format
through the right channel
at the right time in the workflow
Comprehensive Evaluation
Evaluation
Patient Outcomes• Ranging from QoL to morbidity and mortality• Benchmarking and dashboards
Evaluation
Clinician Performance• Process measures• Compliance• Workflow efficiency
Clinician Preference• Usability• Satisfaction
Evaluation Plan (Sepsis Sample)
Emerging Evaluation of CDS
Emerging Evaluation of CDS
TAKEAWAY: Are you comprehensively evaluating the entire process?
1. Speak with providers and developers
2. Evaluate process and patient outcomes
3. Use objective and subjective measures that evaluate preference and performance
What Can You Do Today?
1. Evaluate back-end data
• Identify the problem you are trying to solve
• Identify appropriate targets within data
2. Evaluate front-end design
• Consider basic human factors principles
• Ensure design encourages the preferred action
3. Conduct usability evaluations prior to deployment
• Include true end-users
4. Comprehensively evaluate the entire process
• Evaluate process and patient outcomes
• Use objective and subjective measures of preference and performance
