AHRQ IQI Clinical Validation Panels Sheryl Davies, MA Stanford University

AHRQ IQI Clinical Validation AHRQ IQI Clinical Validation PanelsPanels

Sheryl Davies, MASheryl Davies, MA

Stanford UniversityStanford University

OutlineOutline

PurposePurpose Composition and RecruitmentComposition and Recruitment Rating ProcessRating Process Overall themesOverall themes Indicator resultsIndicator results Take home lessonsTake home lessons

Purpose of the Clinical PanelsPurpose of the Clinical Panels

IQIs and PQIsIQIs and PQIs– Developed 1999-2001Developed 1999-2001– Based on established indicatorsBased on established indicators– Did not undergo panel reviewDid not undergo panel review

Panel review establishes face validity of the Panel review establishes face validity of the indicatorsindicators

Standardize available evidence for all AHRQ Standardize available evidence for all AHRQ QIsQIs– Establish face validity for one stakeholder groupEstablish face validity for one stakeholder group– Update evidenceUpdate evidence

Panel review for IQIs considered for NQF Panel review for IQIs considered for NQF endorsementendorsement

MethodsMethods

Modified RAND/UCLA Appropriateness Method Modified RAND/UCLA Appropriateness Method (Nominal Group Technique)(Nominal Group Technique)

Physicians of various specialties/subspecialties Physicians of various specialties/subspecialties and other health professionals were recruited and other health professionals were recruited with the assistance of relevant organizationswith the assistance of relevant organizations

22 contacted organizations nominated 22 contacted organizations nominated – 103 clinicians nominated 103 clinicians nominated 60 accepted 60 accepted 45 45

eligible eligible

Panelists selected in order to form diverse panelsPanelists selected in order to form diverse panels

PanelistsPanelists

Female: 18%Female: 18% Academic: 71%Academic: 71% Geographic Geographic

– East: 44%East: 44%– West: 29%West: 29%– Other: 27%Other: 27%

Practice settingPractice setting– Urban: 67%Urban: 67%– Suburban: 42%Suburban: 42%– Rural: 18%Rural: 18%

Funding of primary hospital:Funding of primary hospital:– Private: 44%Private: 44%– Public: 27%Public: 27%

Underserved patient population: 56%Underserved patient population: 56%

Panel methods: RatingsPanel methods: Ratings

Initial ratingsInitial ratings– Packet of information summarizing evidencePacket of information summarizing evidence– Approx. 10 questionsApprox. 10 questions

Tailored to the indicator typeTailored to the indicator type 9 point scale9 point scale Overall usefulness for quality improvement, comparative Overall usefulness for quality improvement, comparative

reportingreporting– Compiled ratings provided to panelistsCompiled ratings provided to panelists

Conference callConference call– Discuss differencesDiscuss differences– Consensus on definition changesConsensus on definition changes

Final ratingsFinal ratings– Empirical analyses providedEmpirical analyses provided– Using same questionnaire as initial ratingsUsing same questionnaire as initial ratings

Results: Overarching themesResults: Overarching themes

Case mix variabilityCase mix variability ReliabilityReliability Volume measures as indirect measures Volume measures as indirect measures

of qualityof quality– Composite measuresComposite measures

Cardiac PanelCardiac Panel

Reviewed 5 indicators: Reviewed 5 indicators: – AAA Volume/MortalityAAA Volume/Mortality– Pediatric Heart Surgery Volume/MortalityPediatric Heart Surgery Volume/Mortality– Bilateral CatheterizationBilateral Catheterization

11 clinicians: vascular surgeons, 11 clinicians: vascular surgeons, pediatric cardiologists, pediatric pediatric cardiologists, pediatric cardiovascular surgeons, interventional cardiovascular surgeons, interventional cardiologists, pediatric ICU nurse, cardiologists, pediatric ICU nurse, surgical nursesurgical nurse


Indicator Overall – QI Overall - Comparative

AAA Mortality Acceptable (7) Unclear (6)

AAA Volume Acceptable (7) Acceptable (7)

Bilateral catheterization

Unclear (5) Unclear with disagreement (5)

Pediatric heart surgery volume

Acceptable (8) Acceptable (8)

Pediatric heart surgery mortality






• Case mix variability

• Ruptured vs. unruptured; endovascular vs. open

• Bias: Slight overadjustment for endovascular (12%) and underadjustment for ruptured (12%)

• Total volume (ruptured and unruptured) best predictor of outcomes

• Stratify by surgical approach (endovascular vs. open)






• Ruptured vs. unruptured; endovascular vs. open

• Bias: Slight overadjustment for endovascular (12%) and underadjustment for ruptured (12%)

• Total volume best predictor of outcomes

• Stratify by surgical approach (endovascular vs. open)



Pediatric heart surgery volume


Pediatric heart surgery mortality



• Supported use of RACHS

• Correlations of hospital volume for each RACHS complexity are robust (r = 0.74 – 0.95)

• Best predictor of outcome is total volume, rather than by complexity



Bilateral catheterization

Unclear (5) Unclear w/ disagreement (5)

• Modification: Expand list of appropriate indications for bilateral catheterization

• Primarily a resource indicator

• Charting of indications may be poor

• May result in decrease of appropriate uses

Surgical Resection PanelSurgical Resection Panel

Reviewed 4 indicators:Reviewed 4 indicators:– Esophageal Resection Volume/MortalityEsophageal Resection Volume/Mortality– Pancreatic Resection Volume/MortalityPancreatic Resection Volume/Mortality

13 clinicians: thoracic surgeons, general 13 clinicians: thoracic surgeons, general surgeons (including GI and oncology), surgeons (including GI and oncology), oncologists, internist, oncologists, internist, gastroenterologists, surgical nursegastroenterologists, surgical nurse



Esophageal resection mortality Acceptable (7) Acceptable (7)

Esophageal resection volume Acceptable (7) Acceptable (7)

Pancreatic resection mortality Acceptable (7) Acceptable (7)

Pancreatic resection volume Acceptable (7) Acceptable (7)


Esophageal resection • Case mix variability

• Risk adjustment performs well. Underestimates risk for patient with middle esophageal and unspecified site cancer (22%, 37%).

Pancreatic resection• Case mix variability

• Over-estimates risk for total pancreatectomy (68%), lesser extent underestimates risk for Whipple (15%). Issue raised with 3M.

• Low rates

Geriatric PanelGeriatric Panel

Reviewed 4 indicators:Reviewed 4 indicators:– Acute Stroke MortalityAcute Stroke Mortality– Hip Fracture MortalityHip Fracture Mortality– Hip Replacement MortalityHip Replacement Mortality– Incidental AppendectomyIncidental Appendectomy

14 clinicians: internists (including geriatrics 14 clinicians: internists (including geriatrics and hospital medicine), neurologists, general and hospital medicine), neurologists, general surgeon, interventional radiologist, orthopedic surgeon, interventional radiologist, orthopedic surgeons, neurosurgeon, diagnostic surgeons, neurosurgeon, diagnostic radiologist, nurse, physical therapistradiologist, nurse, physical therapist



Acute stroke mortality

Unclear (6.5) Unclear with disagreement (5)

Incidental appendectomy

Acceptable (7) Unclear (6)

Hip fracture mortality Acceptable (7) Acceptable (7)

Hip replacement mortality

Unclear due to disagreement (7)

Unclear (6)










Unclear (6)

• Case mix variability: Stroke type (hemorrhagic, ischemic, subarachnoid)

• Risk adjustment accounts for almost all difference in risk

• Patient factors such as delay in presenting for care










Unclear (6)

• Exclude patients with hip fracture


• Risk adjustment somewhat overestimates risk for revision

• Rates very low, reliability concerns










Unclear (6)

• “Is it still being done?”

• “If it is still being done, it shouldn’t be done. Then it is a good indicator”

• “I am having a hard time getting excited about this indicator”










Unclear (6)

• Limit to the elderly


• Risk adjustment accounts for both repair type and fracture location

ConclusionConclusion

Overall good reception of indicatorsOverall good reception of indicators Recommendations considered in context Recommendations considered in context

of other validation effortsof other validation efforts Indicator revisions implemented in Indicator revisions implemented in

February 2008February 2008 Further efforts to improve indicators or Further efforts to improve indicators or

develop additional evidence develop additional evidence

Questions?Questions?

Documents

AHRQ IQI Clinical Validation Panels Sheryl Davies, MA Stanford University