Diagnostic Accreditation Program A Systematic Approach to Quality and Safety in Diagnostics with emphasis on Medical Peer Review Dr. Carlow, MD CCFP

Diagnostic Accreditation Program

A Systematic Approach to Quality and Safety in Diagnostics with emphasis on Medical Peer Review

Dr. Carlow, MD CCFP

Objectives

To describe why this is important

To identify what is being done throughout healthcare

To define key principles and practices of a systematic approach

To define issues and solutions for two diagnostic modalities including examples from the field

Why is this important?

• An elderly man underwent chemotherapy for GI cancer at BCCA in the early 1990’s. In error, he received 10 times the normal dose of 5 FU and died as a consequence.

• A Boston health reporter in her late 30’s received a large overdose of chemotherapy for breast cancer at the Dana Farber Cancer Institute and died.

• Both of these tragic events led to major systematic changes.


• A 34 year old woman diagnosed with neuroendocrine cancer had five surgeries to exise a cyst, remove lower jaw and teeth, and undergo facial reconstruction. Her slides were contaminated by cells from another patient. She did not have cancer.

• A patient with a positive pregnancy test underwent pelvic ultrasound. The uterus was described as empty. The patient received methotrexate for the treatment of ectopic pregnancy. On review, another radiologist subsequently diagnosed normal intrauterine pregnancy.


Diagnostic Errors in the daily News

• Pathology errors force thousands to be retested in New Brunswick,

G & M Feb 08

• Disgraced Ontario pathologist says errors not all his fault

G & M Mar 08

• Serious quality-control problems plague hospital labs in Canada

G & M Mar 08

• 108 women died after botched cancer tests Newfoundland says

G & M Mar 08• Errors found in work of another pathologist (6% error rate)

G & M May 08


More than Anecdotes

• Harvard medical practices study (Leape NEJM 91) 3.7% with disabling injuries caused by medical treatment.

• Institute of Medicine (U.S.) report released in 1999 – To Err is Human: Building a Safer Health System

• 44,000 to 98,000 preventable deaths

• Canadian adverse events study: • Adverse event rate of 7.5 per 100 hospital admissions

Baker et al CMAJ 2004


More than Anecdotes

• American physicians got it right 55% of the time

McGlynn et al NEJM 2003

• Many studies reveal significant variations in practice/low rates of standardization

• Swiss Cheese


What about diagnostic errors?

• What types of medical errors occur more frequently – medication or diagnostic errors?

• In two recent studies of malpractice claims – diagnostic errors far outnumbered medication errors as a cause for claims

• Diagnostic errors are underemphasized and understudied


What about autopsy discovered errors?

• Multi decade study

Shojania et al

JAMA, 2003• Median error rate 23.5% for major errors

• Although error rates have declined over the decades, rates are sufficiently high enough that ongoing use is warranted

• U.S. national average autopsy rate is 5%

What is being done?

• Agency for Health Care Research & Quality (AHRQ)• Canadian Patient Safety Institute (CPSI)• Institute for Health Care Improvement (IHI)• Greater expectations from standard setting bodies:

• CCHSA• JCAHO• CAP• ACR

What is being done?

• Safer health care now• Collaboratives• 5 million lives campaign• Hospital standardized mortality ratio (HSMR)• Global Trigger Tool

What is being done?

• Quality networks • Provincial councils on Quality and Safety• Governing Board’s focusing more on quality and safety• Standards of professional bodies• Recertification

What is being done?

Organizational Initiatives

• Veterans Administration/Kaiser Permanente• BC Cancer Agency

These are two examples of the systematic application of evidence and the integration of quality and safety.

What is being done?

• Chronic Disease Management Initiatives• Hypertension• CHF• Diabetes

• Evidence based stroke program• Campbell River hospital

What is being done?

What has been learned about the major attributes of a systematic approach?

• Fragmented and isolated initiatives are quite pervasive and ineffective

• Importance of research driven evidence based care• The important role of clinical decision support systems

and tools

What is being done?

• The integration of quality and safety• The importance of overall system design and clinical

governance:• e.g. Trauma system• e.g. micro systems

• Thorough knowledge of improvement methods and tools including:

• Knowledge of processes• Quality improvement cycles• Root cause analysis• Rapid cycle improvement

What is being done?

• Quality planning and priorities• A clearly set out agenda for quality and safety

• An enabling culture• Leadership commitment• Professional responsibility• Inter-professional collaboration• Non-punitive reporting• Disclosure• Improvement mindset• Thinking and acting as a system• Accountability• Breakthrough thinking/aggressive targets

What is being done?

• Surveillance/Monitoring/Measurement of processes, outcomes and benchmarking

• Quality and Safety infrastructure support• Technological support

Standards of professional bodies are now reflecting these attributes

What is being done?

Causes of Error

• Variation in practice with variable inputs• Complexity – too many steps• Inconsistent knowledge, training and language

(terminology)• Human factors in routine repetitive tasks• Deadlines/stress/excessive workload• Handoffs – transfer of information• Cultural issues – lack of openness and freedom of

expression• Unsystematic/adhoc approaches

• Swiss cheese effect

How should we proceed in Diagnostic Services?

• Do we know enough about the various attributes of diagnostic errors?

• Are not traditional methods of medical peer review adequate and working well?


Areas that need attention

• Better definition of what constitutes an error• Greater consistency in definitions, terminology and

standardization of reporting• Better tools to assess significance of errors


Areas that need attention

• More research on the extent of errors and their causes• More research on the relationship between errors and

adverse affects• Being clearer about acceptable rates of errors


Traditional methods of peer review

• Morbidity and mortality conferences• Autopsy• Malpractice claims analysis• Error reporting systems


Traditional methods of peer review

• Chart review• Observation of patient care• Clinical surveillance• Administrative data analysis• Electronic medical record review


Many of these have positive attributes, however:• Low case numbers• Hind sight bias• Under reporting• Absence of standardization• Some have a linkage to total organizational effort• Some not specific enough for program or department

Anatomic Pathology Errors


• In general anatomic diagnoses are highly accurate?• In the opinion of several, errors are not rampant• Diagnostic variation is not uncommon, but not all harmful• Depends on what are acceptable results

• Medical quality affected by all phases of the system


Life Cycle Data

• Data indicates the importance of gathering information over the whole testing cycle

• Carroro et al in Clinical Chemistry 2007 report• 61.9% pre analytic errors• 15% analytic• 23% post analytic


Pre Analytic Phase

• In this phase of the test cycle the problems more frequently relate to:

• Specimen I.D.• Sample quality• Availability of clinical information


Pre Analytic Phase

• In one large study 6% of cases were defective at accessioning with defective I.D. as the 2nd largest category

Nakhleh et al CAP Q probes

APLM 1996• A survey of 341 labs revealed no clinical history in 2.4% of cases.

When corrected – change in diagnoses in 6.1% of cases

Nakhleh et al CAP Q-probes

APLM 1998


Pre Analytic Phase

• Patient I.D. errors in SP are the most rapidly growing category of malpractice claims in the U.S. Most involve switch of specimens and most involve needle biopsy of prostate and breast


Pre Analytic Phase

• Errors in thyroid gland FNA with relatively high false positive and false negative rates – quality of tissue sampling by non-pathologists

• FNA\histologic correlation reveals ¼ of thyroid cancer patients are misdiagnosed as not having cancer due to:

• Errors in specimen quality• Misinterpretation

Raab et al

ASLP 2006


Analytic Phase

• In a 4 hospital review up to 12% of tissues examined by pathologists resulted in errors, more than 1/3 were associated with harm (AHRQ funded)

Raab, Cancer 2005

• Up to 15% of patients with lung mass misdiagnosed due to pathology errors, different rates among hospitals due to “Big Dog” effect and using different methods


Analytical Phase

• Average discrepancy frequency in pathology reports from74 labs on secondary review is 6.7% with 5% of these having an affect on patient care (1% of all cases)

• Canadian Pathology Error Rates:

• Retro 14.1% overall rate1.2% major

• Prospect 13% overall1.7% major

Lind 1995AJSP


Analytic Phase

• Most studies are single institution hence variation• However multi institutional studies reveal a discrepancy

rate of 6.7% with between 1 and 1.7% causing harm• What is an acceptable level of performance?


Analytic Phase

• Consider that a 1% error rate equates to 10,000 errors per million

• Industrial six sigma standard is 3.4 defects per million• Industry average – four sigma = 6210 defects per million• Should a six sigma standard apply to pathologists?


Post Analytic Phase

• Two aspects of the post analytic phase that are the most important: completeness of reporting; 28.4% increase in complete

reporting using computer based synoptic reports Communication of critical results and customization of critical

values for each institution


Errors

Medical Quality Improvement is most effective if collection, processing, interpretation and connection to care providers are considered as an integrated system


Solutions/Tools

Have a plan and priorities for quality improvement and safety, consider: Health Authority priorities Standards of professional bodies Guidance in literature Performance data Internal assessment – Process map

Identify priority projects Mission Culturally aligned teams Improvement methods (PDCA) and root cause analysis, lean design 6 sigma

leap frog


Solutions/Tools

Generic Laboratory Test Cycle Phases

Test Request Report Interpretation

Procedural

Patient and specimen preparation, identification, transportation, handling,

accession

Technical & Diagnostic

Test method, lab protocols, criteria, terminology,

accuracy, report content, analytic timelines

Communication

Report delivery, format, clarity,

overall timeliness, integration of

information, satisfaction

Preanalytic Analytic Postanalytic


Error types and test-cycle phases.

Solutions/Tools


Solutions/Tools

• Standardization• Terms, language, processes, tasks – work is to be done in a

certain way• Adopting standardized, structured, synoptic reporting formats

province wide• Consider computerized capture of structured data/synoptic

reports linked to databases allowing best practice comparisons, information distribution, trend analysis and discrepancy identification

• e.g. mTuitive


Solutions/Tools

• Peer review• Blinded unbiased double slide review, selecting areas of high

risk for error• Amended reports are decreased with 2nd pathologist review

Nakhleh et alAPLM 1998

• Prostate cancer – impact of 2nd pathologist on Gleason score:• 25.2% change• 14.8% change in management

Thomas et alBrachytherapy 2007


Solutions/Tools

• Double viewing dilemma: Is error reduction frequency sufficiently high to warrant the effort?

• Consider digital pathology system (e.g. ScanScope)• Digitize slides• Desktop computer viewing• Multiple viewer conferencing• Can Link through telepathology to remote locations/single pathologists• Improved turnaround and better use of path times• Can correlate slides with CT and MRI scans


Solutions/Tools

• Frozen/permanent section, discordant, monitoring – sustained improvement in performance

Raab et al

ADLM 2006• FNA / Histologic correlation

• Toyota production system redesign – standard terminology and immediate interpretation

• Fewer diagnostic errors

Raab et al

ASCP 2006• Improving skill / concentrating expertise in FNA

Cytology/Histologic correlation Q tracks program showed improvement in pap. smear performance in preanalytic sampling

Raab et al

APLM Jan 08


Solutions/Tools

• Adopt a system for measuring performance of key processes• e.g. IQLM (U.S.) – 12 core indicators to evaluate lab quality

• Participation in cooperative programs access multiple institutions/databases:

• Q-Tracks• Q-Probes

• System wide approach to reporting critical values• Conference, random, focused, amended report, tumour board

reviews


Solutions/Tools

• Improve access to clinical information• Electronic Medical Record• Better defined linkages to a large variety of clinical microsystems

(users)• A culture that supports change

• Teamwork• Willingness to challenge each other• Acknowledging error in a non-punitive way• Sharing performance information

• Knowledgeable well trained staff• Departmental CME• Education in QI and safety methods

Diagnostic Imaging Errors

• Technological and manpower factors influence quality in diagnostic imaging

• Radiology’s Achilles heel:• Error and variation in the interpretation of the Roentgen Image, now the

weakest aspect of clinical imaging

Robinson, St. James UH

Leads, UK 1997


• Observations of Henry Garland in 1959:• 30% of chest radiographs that are positive for disease will be missed• Awakened the profession to the extent of errors

• But have things changed?• Goddard et al BJR 2001 – little change in past 50 years• Internal error rate by same radiologist can be as high as 25% - 30%


• Shively – Imaging economics 2003:• Many could be avoided if a simple protocol followed• Errors in stroke CT fell from 15% to 1%

• Shriger, JAMA 1998• 49% of radiologists reading CT Scans as part of a large study missed at

least 1 stroke


• Turkington et al PMJ• 14 out of 57 cases of confirmed lung cancer missed• Delays in diagnosis and treatment

• Non-radiologists in emergency departments – rate of misinterpreted radiographs is high (many studies) 20-25% for CT scans


• Kruskal, Radiology 2006• On-line quality assurance reporting system – Beth Israel (Harvard) –

329 cases in 9 months• Communication errors 18%• Interpretation errors 20%• Missed diagnoses 30%• Procedural complications 16%

• Renfrew – Radiology 1992• 182 reported errors• 126 perceptual• 56 mishaps


• Washington Post 2006 based a study by U.S. pharmacopeia• Medication errors that cause harm are 7 times more frequent in radiology

departments than in other hospital settings


Reasons for Error

• Failure to consult old reports• Incomplete clinical history• Failure to suggest next appropriate procedure• Technique limitations


Reasons for Error

• Knowledge problems• Errors in interpretation• Errors in perception

• Failure to communicate in a timely or clinically appropriate manner

• Interpretation by non-radiologists


Reasons for Error

• Quality performance bar, Lau BIIJ 2007

Access Workload Accuracy TAT

Access Workload Accuracy TAT

• Interlinked

• Output pie is only so big

• Increased expectations compromise accuracy


Solutions/Tools

Have a plan and priorities for quality improvement and safety, consider: Health Authority priorities Standards of professional bodies Guidance in literature Performance data Internal assessment – Process map

Identify priority projects Mission Culturally aligned teams Improvement methods (PDCA) and root cause analysis, lean

design 6 sigma leap frog


QUALITY MAP

Patient Global Outcome

Physician Patient Exam

Appropriateness

Reviews finding/

treats patient

Orders Test

Access Finalization Times

Radiology Department Schedules

Waiting Times

Standard Protocol

Exam Performed

Performance Outcomes

Patient Satisfaction

Interpretation

Structured Report

Radiologist Protocol Selection Finalization

Solutions/Tools


Radiology scorecard. Each quality metric from the quality map (Figure 1) and key safety metrics arelisted in the left-hand column. Departmental divisions and operational groups are listed in the top row. Metricsare provided for each box in the scorecard, and the box is color coded (green, yellow, and red) dependingupon operational performance. Practice problems can be quickly identified using this tool.

Solutions/Tools


Solutions/Tools

• Examples:• Knowledge of history and clinical findings – EMR• Careful selection of radiological investigation and linkage to

clinical protocols• A process to ensure comparisons with previous studies• Improvement in working conditions and available time


Solutions/Tools

• Examples:• A process for review and timely follow-up on all discrepancies

on images ordered by and interpreted by non-radiologists• Development of a quality and safety performance

reporting/monitoring system• Develop targeted areas for prospective clinical surveillance to

identify areas needing improvement• Address potential for medication incidents


Solutions/Tools

• Structured anatomic/region specific reports – technology assisted• Voice automated• Structured report templates• Sensitive to clinical requirements• Standardization/consistency• Reduces transcription errors• Faster TAT• Improves report clarity• Linkage to database• Facilitates peer review


Solutions/Tools

Peer review, consider: 5% review mandated by ACR. Will small sample size enable

valid individual or departmental reviews? SMPBC

• False negatives identified through linkage to cancer registry• Feedback to program leaders and individual radiologists


Solutions/Tools

• Technology enabled peer review built into work routine (e.g. RADPEER)

• Software enable second review of past reports/films to be submitted on electronic format

• Can evaluate past reports – scoring system• Can be done rapidly• Central data bank for peer comparisons, departmental reviews and

individual reviews• Meaningful data

• International Radiology Quality Network

Pathology/Diagnostic Imaging Clinical Integration

• Closer interaction between clinicians and those in diagnostic services is associated with better outcomes

• The development of clinical protocols and structured diagnostic reports can enable requirements of each to be addressed

• Consider the clinical microsystem to strengthen engagement with clinical care teams

Health System as an inverted Pyramid

Mesosystem• departments

•Programs

CCU

MacrosystemSenior Leaders

Patients & Family Needs

Stroke ICU Renal ED

Board

• Clinical evidence base

• System support

• Clinical quality measures

Sharp end

Blunt end

Quality by Design Batalden

Clinical Microsystem

Diagnostic Services and Clinical Microsystems

Diagnostic Services

Clinical Microsystems

• Information transfer

• Participation in clinical requirements

• Integration

• Coordination

Summary/Conclusions

• Diagnostic Services• Consider QI and safety as part of a system• Have a plan with priorities• Address cultural barriers• Develop knowledge in QI methods/tools/root cause analysis• Develop performance measurement• Enhance clinical integration• Push for technology

Summary/Conclusions

Provincial Policy and Health Authorities Enabling technology Redesign/reengineer the system Capital equipment planning

• DAP• Peer review standards• Surveyor preparation/survey tools• Facilitate sharing/best practice dissemination

Documents

Diagnostic Accreditation Program A Systematic Approach to Quality and Safety in Diagnostics with emphasis on Medical Peer Review Dr. Carlow, MD CCFP