Panel Reviews of the AHRQ QIs: AHRQ and NQF panels Sheryl Davies, MA Stanford University

Panel Reviews of the AHRQ Panel Reviews of the AHRQ QIs: AHRQ and NQF panelsQIs: AHRQ and NQF panels

Sheryl Davies, MASheryl Davies, MA

Stanford UniversityStanford University

OutlineOutline

PurposePurpose AHRQ Panel Review MethodsAHRQ Panel Review Methods Example of the IQI Panel ReviewExample of the IQI Panel Review NQF Review of the QIsNQF Review of the QIs Indicators receiving NQF approvalIndicators receiving NQF approval Take home lessonsTake home lessons

Purpose of the Clinical PanelsPurpose of the Clinical Panels

Panel review establishes face validity of Panel review establishes face validity of the indicatorsthe indicators

Refine definitions of the indicatorsRefine definitions of the indicators Standardize available evidence for all Standardize available evidence for all

AHRQ QIsAHRQ QIs– Establish face validity for one stakeholder Establish face validity for one stakeholder

groupgroup– Update evidenceUpdate evidence

MethodsMethods

Modified RAND/UCLA Appropriateness Method Modified RAND/UCLA Appropriateness Method (Nominal Group Technique)(Nominal Group Technique)

Physicians of various specialties/subspecialties Physicians of various specialties/subspecialties and other health professionals were recruited and other health professionals were recruited with the assistance of relevant organizationswith the assistance of relevant organizations

Panelists selected in order to form diverse panelsPanelists selected in order to form diverse panels– Male (80%), academic (71%), Geographic, Rural Male (80%), academic (71%), Geographic, Rural

(18%), Underserved patient population (50%)(18%), Underserved patient population (50%)

Panel methods: RatingsPanel methods: Ratings

Initial ratingsInitial ratings– Packet of information summarizing evidencePacket of information summarizing evidence– Approx. 10 questionsApprox. 10 questions

Tailored to the indicator typeTailored to the indicator type 9 point scale9 point scale Overall usefulness for quality improvement, comparative Overall usefulness for quality improvement, comparative

reportingreporting– Compiled ratings provided to panelistsCompiled ratings provided to panelists

Conference callConference call– Discuss differencesDiscuss differences– Consensus on definition changesConsensus on definition changes

Final ratingsFinal ratings– Empirical analyses providedEmpirical analyses provided– Using same questionnaire as initial ratingsUsing same questionnaire as initial ratings

Results: Overarching themesResults: Overarching themes

Three panels reviewed IQIs prior to NQF reviewThree panels reviewed IQIs prior to NQF review– Cardiac, Geriatric, Cardiac, Geriatric,

Case mix variabilityCase mix variability ReliabilityReliability Volume measures as indirect measures of qualityVolume measures as indirect measures of quality

– Composite measuresComposite measures

Cardiac PanelCardiac Panel

Reviewed 5 indicators: Reviewed 5 indicators: – AAA Volume/MortalityAAA Volume/Mortality– Pediatric Heart Surgery Volume/MortalityPediatric Heart Surgery Volume/Mortality– Bilateral CatheterizationBilateral Catheterization

11 clinicians: vascular surgeons, 11 clinicians: vascular surgeons, pediatric cardiologists, pediatric pediatric cardiologists, pediatric cardiovascular surgeons, interventional cardiovascular surgeons, interventional cardiologists, pediatric ICU nurse, cardiologists, pediatric ICU nurse, surgical nursesurgical nurse


Indicator Overall – QI Overall - Comparative

AAA Mortality Acceptable (7) Unclear (6)

AAA Volume Acceptable (7) Acceptable (7)

• Case mix variability

• Ruptured vs. unruptured; endovascular vs. open

• Bias: Slight overadjustment for endovascular (12%) and underadjustment for ruptured (12%)

• Total volume (ruptured and unruptured) best predictor of outcomes

• Stratify by surgical approach (endovascular vs. open)



AAA Mortality Acceptable (7) Unclear (6)

AAA Volume Acceptable (7) Acceptable (7)


• Ruptured vs. unruptured; endovascular vs. open

• Bias: Slight overadjustment for endovascular (12%) and underadjustment for ruptured (12%)

• Total volume best predictor of outcomes

• Stratify by surgical approach (endovascular vs. open)



Pediatric heart surgery volume

Acceptable (8) Acceptable (8)

Pediatric heart surgery mortality

Acceptable (8) Acceptable (8)


• Supported use of RACHS

• Correlations of hospital volume for each RACHS complexity are robust (r = 0.74 – 0.95)

• Best predictor of outcome is total volume, rather than by complexity



Bilateral catheterization

Unclear (5) Unclear w/ disagreement (5)

• Modification: Expand list of appropriate indications for bilateral catheterization

• Primarily a resource indicator

• Charting of indications may be poor

• May result in decrease of appropriate uses

Surgical Resection PanelSurgical Resection Panel

Reviewed 4 indicators:Reviewed 4 indicators:– Esophageal Resection Volume/MortalityEsophageal Resection Volume/Mortality– Pancreatic Resection Volume/MortalityPancreatic Resection Volume/Mortality

13 clinicians: thoracic surgeons, general 13 clinicians: thoracic surgeons, general surgeons (including GI and oncology), surgeons (including GI and oncology), oncologists, internist, oncologists, internist, gastroenterologists, surgical nursegastroenterologists, surgical nurse

Surgical Resection PanelSurgical Resection Panel


Esophageal resection mortality Acceptable (7) Acceptable (7)

Esophageal resection volume Acceptable (7) Acceptable (7)

Pancreatic resection mortality Acceptable (7) Acceptable (7)

Pancreatic resection volume Acceptable (7) Acceptable (7)

Geriatric PanelGeriatric Panel

Reviewed 4 indicators:Reviewed 4 indicators:– Acute Stroke MortalityAcute Stroke Mortality– Hip Fracture MortalityHip Fracture Mortality– Hip Replacement MortalityHip Replacement Mortality– Incidental AppendectomyIncidental Appendectomy

14 clinicians: internists (including geriatrics 14 clinicians: internists (including geriatrics and hospital medicine), neurologists, general and hospital medicine), neurologists, general surgeon, interventional radiologist, orthopedic surgeon, interventional radiologist, orthopedic surgeons, neurosurgeon, diagnostic surgeons, neurosurgeon, diagnostic radiologist, nurse, physical therapistradiologist, nurse, physical therapist



Acute stroke mortality

Unclear (6.5) Unclear with disagreement (5)

Incidental appendectomy

Acceptable (7) Unclear (6)

Hip fracture mortality Acceptable (7) Acceptable (7)

Hip replacement mortality

Unclear due to disagreement (7)

Unclear (6)










Unclear (6)

• Case mix variability: Stroke type (hemorrhagic, ischemic, subarachnoid)

• Risk adjustment accounts for almost all difference in risk

• Patient factors such as delay in presenting for care










Unclear (6)

• Exclude patients with hip fracture


• Risk adjustment somewhat overestimates risk for revision

• Rates very low, reliability concerns










Unclear (6)

• “Is it still being done?”

• “If it is still being done, it shouldn’t be done. Then it is a good indicator”

• “I am having a hard time getting excited about this indicator”










Unclear (6)

• Limit to the elderly


• Risk adjustment accounts for both repair type and fracture location

National Quality Forum ReviewNational Quality Forum Review

Provides additional evidence of face Provides additional evidence of face validityvalidity

Another outside evaluation of available Another outside evaluation of available evidenceevidence

Usefulness for comparative reporting as Usefulness for comparative reporting as well as quality improvementswell as quality improvements

Suggested potential improvements to Suggested potential improvements to indicatorsindicators

NQF Endorsed IndicatorsNQF Endorsed Indicators

IQIsIQIs– Esophageal Resection Volume (#01)*Esophageal Resection Volume (#01)*– Pancreatic Resection Volume (#02)*Pancreatic Resection Volume (#02)*– Abdominal Aortic Aneurysm (AAA) Repair Volume (#04)*Abdominal Aortic Aneurysm (AAA) Repair Volume (#04)*– Esophageal Resection Mortality (#08)*Esophageal Resection Mortality (#08)*– Pancreatic Resection Mortality (#09)*Pancreatic Resection Mortality (#09)*– Abdominal Aortic Aneurysm (AAA) Repair Mortality (#11)*Abdominal Aortic Aneurysm (AAA) Repair Mortality (#11)*– AMI Mortality (#15 and #32)AMI Mortality (#15 and #32)– CHF Mortality (#16)*CHF Mortality (#16)*– Acute Stroke Mortality (#17)*Acute Stroke Mortality (#17)*– Hip Fracture Mortality (#19)*Hip Fracture Mortality (#19)*– Pneumonia Mortality (#20)*Pneumonia Mortality (#20)*– Incidental Appendectomy in the Elderly (#24)*Incidental Appendectomy in the Elderly (#24)*– Bi-lateral Catheterization (#25)*Bi-lateral Catheterization (#25)*

*NQF endorsed*NQF endorsed

NQF Endorsed IndicatorsNQF Endorsed Indicators

PDIsPDIs– Accidental Puncture or Laceration (#01)*Accidental Puncture or Laceration (#01)*– Decubitus Ulcer (#02)*Decubitus Ulcer (#02)*– Iatrogenic Pneumothorax (#05)*Iatrogenic Pneumothorax (#05)*– Pediatric Heart Surgery Mortality (#06)*Pediatric Heart Surgery Mortality (#06)*– Pediatric Heart Surgery Volume (#07)*Pediatric Heart Surgery Volume (#07)*– Postoperative Wound Dehiscence (#11)*Postoperative Wound Dehiscence (#11)*– Blood Stream Infection in Neonates (#02)*Blood Stream Infection in Neonates (#02)*

PSIsPSIs– Death among Surgical Inpatients with Treatable Serious Complications Death among Surgical Inpatients with Treatable Serious Complications

(#04)*(#04)*– Iatrogenic Pneumothorax (#06)*Iatrogenic Pneumothorax (#06)*– Postoperative Hip Fracture (#08)Postoperative Hip Fracture (#08)– Postoperative DVT or PE (#12)*Postoperative DVT or PE (#12)*– Postoperative Wound Dehiscence (#14)*Postoperative Wound Dehiscence (#14)*– Accidental Puncture or Laceration (#15)*Accidental Puncture or Laceration (#15)*– OB Trauma with and without Instrument (#18 and #19)OB Trauma with and without Instrument (#18 and #19)– Birth Trauma (#17)* Birth Trauma (#17)*

Reasons Indicators are not NQF Reasons Indicators are not NQF endorsedendorsed

Some indicators not submittedSome indicators not submitted– Needed further development workNeeded further development work– Similar indicators already NQF endorsedSimilar indicators already NQF endorsed

Some indicators withdrawnSome indicators withdrawn– New evidence collected needed further New evidence collected needed further

consideration before completing processconsideration before completing process NQF panel concernsNQF panel concerns

– Preventability and links between process Preventability and links between process and outcomeand outcome

Examples of changes proposed Examples of changes proposed by NQFby NQF

Rare indicators to be expressed as counts Rare indicators to be expressed as counts rather than ratesrather than rates– Transfusion reaction, Foreign BodyTransfusion reaction, Foreign Body

Requirements to use POARequirements to use POA– Decubitus ulcer, Foreign BodyDecubitus ulcer, Foreign Body

Harmonization of measuresHarmonization of measures– Death Among Surgical Inpatients with Death Among Surgical Inpatients with

Complications (Formerly FTR), Birth TraumaComplications (Formerly FTR), Birth Trauma Time Limited EndorsementsTime Limited Endorsements

– Neonatal indicatorsNeonatal indicators

ConclusionConclusion

Overall good reception of indicatorsOverall good reception of indicators Recommendations considered in context Recommendations considered in context

of other validation effortsof other validation efforts Indicator revisions implemented in Indicator revisions implemented in

Future Releases, in addition to coding Future Releases, in addition to coding and changes from user experienceand changes from user experience

Further efforts to improve indicators or Further efforts to improve indicators or develop additional evidence develop additional evidence

Questions?Questions?

Documents

Panel Reviews of the AHRQ QIs: AHRQ and NQF panels Sheryl Davies, MA Stanford University