Evaluating the AHRQ Inpatient Mortality Indicators for Public Reporting in California AHRQ Users Meeting Joseph Parker, PhD Brian Paciotti, PhD Merry Holliday-Hanson,

Evaluating the AHRQ Evaluating the AHRQ Inpatient Mortality Inpatient Mortality Indicators for Public Indicators for Public Reporting in Reporting in CaliforniaCalifornia

AHRQ Users MeetingAHRQ Users Meeting

Joseph Parker, PhDJoseph Parker, PhDBrian Paciotti, PhDBrian Paciotti, PhD

Merry Holliday-Hanson, PhDMerry Holliday-Hanson, PhD

Rockville, MDRockville, MD

September 10, 2008September 10, 2008

AHRQ Inpatient Mortality Indicators (IMIs): AHRQ Inpatient Mortality Indicators (IMIs): Background to CA DecisionBackground to CA Decision

• Per statute, OSHPD should be publishing 9 risk-adjusted Per statute, OSHPD should be publishing 9 risk-adjusted hospital mortality reports per yearhospital mortality reports per year

• Traditional approach to producing reports costly, time-Traditional approach to producing reports costly, time-consuming, fraught with delaysconsuming, fraught with delays

• CA patient discharge data now available only 7 months CA patient discharge data now available only 7 months after end of reporting year (inpatient mortality)after end of reporting year (inpatient mortality)

• State death file (necessary for 30-day mortality) not State death file (necessary for 30-day mortality) not available until 15 months after end of reporting yearavailable until 15 months after end of reporting year

• APR-DRG risk model not ‘black box’ anymoreAPR-DRG risk model not ‘black box’ anymore• POA now incorporated in APR-DRG risk adjustment POA now incorporated in APR-DRG risk adjustment

algorithmalgorithm• Some IMIs have undergone NQF vetting processSome IMIs have undergone NQF vetting process

AHRQ Inpatient Mortality Indicators: AHRQ Inpatient Mortality Indicators: California Plans for Public ReportingCalifornia Plans for Public Reporting

Conditions (7)Conditions (7) Procedures (8)Procedures (8)

Acute StrokeAcute Stroke

Gastrointestinal (GI) HemorrhageGastrointestinal (GI) Hemorrhage

Hip Fracture*Hip Fracture*

AMI AMI

AMI without transfer cases AMI without transfer cases

PneumoniaPneumonia**

Congestive Heart FailureCongestive Heart Failure**

Esophageal Resection*Esophageal Resection*

Pancreatic Resection*Pancreatic Resection*

CraniotomyCraniotomy

Carotid EndarterectomyCarotid Endarterectomy

Percutaneous Transluminal Percutaneous Transluminal Coronary Angioplasty (PTCA)Coronary Angioplasty (PTCA)

Abdominal Aortic Aneurysm (AAA) Abdominal Aortic Aneurysm (AAA) RepairRepair**

Hip ReplacementHip Replacement

Coronary Artery Bypass Graft Coronary Artery Bypass Graft (CABG) Surgery(CABG) Surgery

Indicators NOT planned for release in GoldIndicators NOT planned for release in Gold* * Endorsed by NQF May, 2008Endorsed by NQF May, 2008

Comparison of Traditional OSHPD Mortality Comparison of Traditional OSHPD Mortality Reports* with AHRQ IMIsReports* with AHRQ IMIs

OSHPD Mortality ReportsOSHPD Mortality Reports• 30-day mortality30-day mortality• Link with CA death file data (delay)Link with CA death file data (delay)• Include POA informationInclude POA information• Detailed technical reports Detailed technical reports

accompany resultsaccompany results• Data validation study performedData validation study performed• Considered quality “measures”Considered quality “measures”• No national vetting/endorsementNo national vetting/endorsement

AHRQ IMIsAHRQ IMIs• Inpatient mortalityInpatient mortality• No death file data usedNo death file data used• NowNow include POA information include POA information• Minimum documentation (Tech. Minimum documentation (Tech.

Note with links to AHRQ)Note with links to AHRQ)• Data not formally validated in CAData not formally validated in CA• Considered quality “indicators”Considered quality “indicators”• Six received National Quality Six received National Quality

Forum (NQF) endorsement (Hip Forum (NQF) endorsement (Hip replacement mortality withdrawn)replacement mortality withdrawn)

* Not including CABG report based on clinical data* Not including CABG report based on clinical data

Planned and Published OSHPD Quality Metrics: Planned and Published OSHPD Quality Metrics: Varying Levels of Validity Varying Levels of Validity

CA CABG CA CABG ReportReport

OSHPD OSHPD Traditional Traditional

Reports Reports **

OSHPD OSHPD Benchmark Benchmark

ReportsReports****AHRQ IMIsAHRQ IMIs

AHRQ Volume AHRQ Volume & Utilization & Utilization IndicatorsIndicators

Type of DataType of Data Clinical Clinical RegistryRegistry

Patient Patient Discharge DataDischarge Data




Data Quality ChecksData Quality ChecksExtensive, Extensive, ongoing ongoing changeschanges

Automated; no Automated; no changes past changes past acceptanceacceptance




Medical Chart AuditMedical Chart AuditYearlyYearly

With initial With initial validation studyvalidation study

Limited: CHF, Limited: CHF, None: AAANone: AAA NoneNone NoneNone

Risk Model ReviewRisk Model ReviewYearlyYearly InfrequentInfrequent InfrequentInfrequent

Periodic by Periodic by AHRQAHRQ

Periodic by Periodic by AHRQAHRQ

Expert Panel Expert Panel ReviewReview ContinuousContinuous

With initial With initial validation, TACvalidation, TAC TACTAC

Periodic by Periodic by AHRQ & NQFAHRQ & NQF

Periodic by Periodic by AHRQ & NQFAHRQ & NQF

Principal Source of Principal Source of ValidationValidation

National National STS & STS & associated associated literatureliterature

Initial validation Initial validation study & study & literatureliterature

For CHF, CMS For CHF, CMS validation; for validation; for AAA, literatureAAA, literature

Extensive Extensive literature review literature review & NQF vetting& NQF vetting

Extensive Extensive literature review literature review & NQF vetting& NQF vetting

** Includes two reports produced (heart attack, pneumonia) and two in progress Includes two reports produced (heart attack, pneumonia) and two in progress

**** Two reports in progress (AAA repair & CHF) to be released without a formal validation studyTwo reports in progress (AAA repair & CHF) to be released without a formal validation study

AHRQ Message on IMI ValidityAHRQ Message on IMI Validity

• Providers, policy makers, and researchers can use with inpatient Providers, policy makers, and researchers can use with inpatient data to data to identify apparent variationsidentify apparent variations in the quality of inpatient care in the quality of inpatient care

• Although quality assessments based on administrative data cannot Although quality assessments based on administrative data cannot be definitive, they can be used to be definitive, they can be used to flag potential quality problemsflag potential quality problems and and success stories, which can then be further investigated and studied success stories, which can then be further investigated and studied

• Hospital associations, individual hospitals, purchasers, regulators, Hospital associations, individual hospitals, purchasers, regulators, and policymakers at the local, State, and Federal levels can use and policymakers at the local, State, and Federal levels can use readily available hospital administrative data to readily available hospital administrative data to begin the begin the assessment of quality of careassessment of quality of care

OSHPD Message on IMI ValidityOSHPD Message on IMI Validity

OSHPD views these indicators as potentially useful OSHPD views these indicators as potentially useful starting points for examining hospital quality but starting points for examining hospital quality but does does not regard them as definitive measures of qualitynot regard them as definitive measures of quality. . When this information is carefully considered, with its When this information is carefully considered, with its limitations, limitations, alongside other reliable healthcare provider alongside other reliable healthcare provider informationinformation, it may be helpful to patients and , it may be helpful to patients and purchasers when making decisions about healthcare purchasers when making decisions about healthcare treatment choices. Healthcare providers may also treatment choices. Healthcare providers may also benefit from using this information in quality benefit from using this information in quality improvement activities. improvement activities.

OSHPD Implementation of IMIsOSHPD Implementation of IMIs

• Used 2007 CA patient Discharge Data Used 2007 CA patient Discharge Data • Transformed data elements and values into formats for AHRQ Transformed data elements and values into formats for AHRQ

software (Version 3.2)software (Version 3.2)• POA option utilizedPOA option utilized

– APR-DRG risk scores based only on conditions coded as pre-APR-DRG risk scores based only on conditions coded as pre-existing, not hospital-related complicationsexisting, not hospital-related complications

• Calculated risk-adjusted ratesCalculated risk-adjusted rates– Used APR-DRG risk model with coefficients from CA & NYUsed APR-DRG risk model with coefficients from CA & NY– Logistic regression model (with random hospital effects)Logistic regression model (with random hospital effects)

• Calculated statistical outliersCalculated statistical outliers– No hospital case volume limit appliedNo hospital case volume limit applied– 95% upper and lower CIs95% upper and lower CIs– ““Better” and “Worse than Expected” labels usedBetter” and “Worse than Expected” labels used

2007 CA Statewide California IMI Results2007 CA Statewide California IMI Results

Procedure/ConditionProcedure/Condition # Cases# Cases # Hospitals# Hospitals RateRate # Better# Better # Worse# Worse

Esophageal ResectionEsophageal Resection 190190 5959 6.56.5 00 22

Pancreatic ResectionPancreatic Resection 623623 121121 4.54.5 00 55

CraniotomyCraniotomy 11,42711,427 294294 6.26.2 11 1414

Acute StrokeAcute Stroke 49,91549,915 343343 10.410.4 2020 3333

GI HemorrhageGI Hemorrhage 48,69148,691 358358 2.12.1 11 1313

Hip FractureHip Fracture 23,70023,700 300300 2.42.4 00 1414

PTCAPTCA 52,15252,152 154154 1.31.3 22 1111

Carotid EndarterectomyCarotid Endarterectomy 8,1328,132 238238 0.40.4 11 77

Analyses to Detect Major Biases in IMIs Analyses to Detect Major Biases in IMIs Using Internal DataUsing Internal Data

• Do certain types of hospitals (teaching, public, profit, non-Do certain types of hospitals (teaching, public, profit, non-profit) have worse IMI rates than other types of hospitals?profit) have worse IMI rates than other types of hospitals?

• Do hospitals with a large percentage of DNR, palliative Do hospitals with a large percentage of DNR, palliative care, or SNF patients have worse IMI rates than other care, or SNF patients have worse IMI rates than other hospitals?hospitals?

• Do hospitals with very poor POA coding quality (rarely Do hospitals with very poor POA coding quality (rarely coding complications) have better IMI rates than others?coding complications) have better IMI rates than others?

• Do Do high transfer-intensityhigh transfer-intensity hospitals benefit from use of hospitals benefit from use of inpatient mortality compared to 30-day mortality?inpatient mortality compared to 30-day mortality?

• Does the AHRQ inclusion of POA improve validity of Does the AHRQ inclusion of POA improve validity of original IMI by bringing it closer to a gold standard proxy?original IMI by bringing it closer to a gold standard proxy?

Hospital 2007 IMI Rates by Type of HospitalHospital 2007 IMI Rates by Type of Hospital

p<.05 **p<0.01 ***p<0.001 (ANOVA)p<.05 **p<0.01 ***p<0.001 (ANOVA)

Hospital TypeHospital Type

Non-ProfitNon-Profit InvestorInvestor PublicPublic TeachingTeaching

Number of Hospitals*Number of Hospitals* 191191 9696 6262 2020

IMIsIMIs

Craniotomy***Craniotomy*** 5.45.4 4.74.7 11.711.7 6.96.9

Stroke**Stroke** 10.410.4 9.09.0 12.312.3 10.610.6

GI Hemorrhage**GI Hemorrhage** 2.22.2 1.91.9 33 1.81.8

Hip Fracture*Hip Fracture* 2.32.3 2.82.8 3.33.3 2.22.2

PTCA**PTCA** 1.21.2 1.01.0 1.51.5 1.61.6

Carotid EndarterectomyCarotid Endarterectomy 0.40.4 0.30.3 0.50.5 0.10.1

• Includes all CA hospitals meeting minimum volume thresholds: Includes all CA hospitals meeting minimum volume thresholds:

most IMIs will have fewer hospitals reportingmost IMIs will have fewer hospitals reporting

Effect of DNR, Palliative Care, and SNF Effect of DNR, Palliative Care, and SNF Patients on Hospital PerformancePatients on Hospital Performance

• For each hospital, across all patients, calculated the percent of total For each hospital, across all patients, calculated the percent of total who had:who had:

– DNR coded within 24 hours of admissionDNR coded within 24 hours of admission– ICD-9 code for palliative care (99.7)ICD-9 code for palliative care (99.7)– Source of admission = SNFSource of admission = SNF– Marker for unmeasured severityMarker for unmeasured severity

• Correlated % of patients with DNR, PC, & SNF with 8 IMI ratesCorrelated % of patients with DNR, PC, & SNF with 8 IMI rates• For DNR, PC & SNF, determined which 10% of hospitals had highest For DNR, PC & SNF, determined which 10% of hospitals had highest

coding ratescoding rates– Calculated and compared the IMI rates for the high 10% group Calculated and compared the IMI rates for the high 10% group

with other 90%with other 90%• However, when comparing hospitals in top & bottom 5% of mortality However, when comparing hospitals in top & bottom 5% of mortality

performance, no consistent trend in DNR, PC, and SNF patient performance, no consistent trend in DNR, PC, and SNF patient caseloads (even for CHF & stroke)caseloads (even for CHF & stroke)

Correlations Between Hospital DNR, Palliative Correlations Between Hospital DNR, Palliative Care, and SNF Admission Rates and IMI RatesCare, and SNF Admission Rates and IMI Rates

IMIs # Hospitals

Pearson Correlation

DNR RatePall. Care

RateSNF Admit

Rate

Esophageal ResectionEsophageal Resection 1616 0.080.08 0.370.37 0.050.05

Pancreatic ResectionPancreatic Resection 6060 0.100.10 0.190.19 -0.05-0.05

CraniotomyCraniotomy 159159 -0.13-0.13 0.020.02 -0.05-0.05

Acute StrokeAcute Stroke 273273 0.22***0.22*** 0.21***0.21*** -0.08-0.08

Gastrointestinal HemorrhageGastrointestinal Hemorrhage 292292 0.000.00 0.020.02 0.050.05

Hip FractureHip Fracture 277277 0.100.10 0.080.08 0.040.04

PTCAPTCA 132132 0.010.01 0.100.10 0.070.07

Carotid EndarterectomyCarotid Endarterectomy 170170 0.030.03 -0.01-0.01 -0.01-0.01

Community-acquired Community-acquired pneumonia (OSHPD report)pneumonia (OSHPD report) 354354 .11*.11* 0.13*0.13* ------

*p<.05, **p<.01, ***p<.0001*p<.05, **p<.01, ***p<.0001

Do Hospitals Coding Large Numbers of DNR, Palliative Do Hospitals Coding Large Numbers of DNR, Palliative Care, or SNF Admissions Have Higher IMI Rates?Care, or SNF Admissions Have Higher IMI Rates?

IMIsIMIs

DNRDNR Palliative CarePalliative Care SNF AdmissionsSNF Admissions

HighHigh OtherOther HighHigh OtherOther HighHigh OtherOther

Craniotomy (%)Craniotomy (%) ---- ---- 5.25.2 6.36.3 4.34.3 6.26.2

Stroke (%)Stroke (%) 15.8 15.8 10.010.0 11.911.9 10.110.1 8.88.8 10.510.5

GI Hemorrhage (%)GI Hemorrhage (%) 2.72.7 2.12.1 2.32.3 2.12.1 2.32.3 2.12.1

Hip Fracture (%)Hip Fracture (%) 1.71.7 2.62.6 2.32.3 2.62.6 1.81.8 2.62.6

PTCA (%)PTCA (%) 1.81.8 1.31.3 1.31.3 1.31.3 1.51.5 1.31.3

Carotid Endarterectomy (%)Carotid Endarterectomy (%) 00 0.30.3 0.70.7 0.30.3 1.21.2 0.30.3

Impact of Hospital POA Coding Quality on Impact of Hospital POA Coding Quality on PerformancePerformance

• POA analyses using % cases coded POA=‘yes’ (crude POA analyses using % cases coded POA=‘yes’ (crude measure) and 3M POA coding quality metricmeasure) and 3M POA coding quality metric

– No consistent correlation (direction or strength) between % No consistent correlation (direction or strength) between % hospital cases coded POA=“yes” and IMI rateshospital cases coded POA=“yes” and IMI rates

– No consistent difference between lowest & highest 5% of No consistent difference between lowest & highest 5% of hospitals by mortality rate, on % cases coded POA=“yes”hospitals by mortality rate, on % cases coded POA=“yes”

– Using 3M metric, poor coding hospitals are not over-Using 3M metric, poor coding hospitals are not over-represented in lowest 5% mortality rate hospitalsrepresented in lowest 5% mortality rate hospitals

Do Hospitals with Bad POA Coding Have Do Hospitals with Bad POA Coding Have Better IMI Rates Than Other Hospitals?Better IMI Rates Than Other Hospitals?

IMIsIMIs

Hughes et al. MetricHughes et al. Metric OSHPD MetricOSHPD Metric

OtherOther BadBad OtherOther BadBad

Craniotomy (%)Craniotomy (%) 5.75.7 9.49.4 6.06.0 9.19.1

Stroke (%)Stroke (%) 10.610.6 9.39.3 10.410.4 8.78.7

GI Hemorrhage (%)GI Hemorrhage (%) 2.12.1 2.52.5 2.12.1 2.42.4

Hip Fracture (%)Hip Fracture (%) 2.62.6 2.22.2 2.62.6 2.02.0

PTCA (%)PTCA (%) 1.21.2 1.61.6 1.31.3 3.63.6

Carotid Endarterectomy (%)Carotid Endarterectomy (%) 0.30.3 0.30.3 0.30.3 00

Does the Use of Inpatient Mortality vs. Does the Use of Inpatient Mortality vs. 30-Day Mortality Bias Hospital Results?30-Day Mortality Bias Hospital Results?

• 30-day mortality is the preferred measure for outcomes assessment30-day mortality is the preferred measure for outcomes assessment– Not impacted by hospital discharge practicesNot impacted by hospital discharge practices– May result in less timely reportsMay result in less timely reports

• Transfer rates vary greatly by hospitalTransfer rates vary greatly by hospital– Transfers to acute care (excluded in IMIs):Transfers to acute care (excluded in IMIs):– Transfers to SNF/other care: Transfers to SNF/other care:

• MethodsMethods– Created hospital “transfer intensity” measureCreated hospital “transfer intensity” measure

• Rates created for top 10%, middle 80%, and bottom 10% of hospitals Rates created for top 10%, middle 80%, and bottom 10% of hospitals according to ALL patient discharges to non-hospital care (SNF/intermediate according to ALL patient discharges to non-hospital care (SNF/intermediate care, other care)care, other care)need to describe/research betterneed to describe/research better

– Calculated crude inpatient and 30-day mortality rates for hospitals by Calculated crude inpatient and 30-day mortality rates for hospitals by transfer intensity grouptransfer intensity group

– Calculated difference between hospital inpatient and 30-day crude mortality Calculated difference between hospital inpatient and 30-day crude mortality rates by IMI rates by IMI

– Calculated an “impact measure that Calculated an “impact measure that describes to what degree different transfer describes to what degree different transfer intensity groups are favorably impacted by use of inpatient mortalityintensity groups are favorably impacted by use of inpatient mortality

IMIIMI Transfer IntensityTransfer Intensity # Hospitals# Hospitals Inpatient RateInpatient Rate 30-Day Rate30-Day Rate DifferenceDifference ImpactImpact**

CraniotomyCraniotomy HighHigh 66 14.314.3 19.019.0 4.74.7 +2.2+2.2

ModerateModerate 121121 9.99.9 12.812.8 2.92.9 ----

LowLow 1111 8.58.5 10.310.3 1.81.8 -1.1-1.1

StrokeStroke HighHigh 1818 7.27.2 13.713.7 6.56.5 -0.6-0.6


LowLow 1616 10.110.1 14.114.1 4.04.0 -2.9-2.9

GI GI HemorrhagHemorrhagee

HighHigh 2525 2.52.5 4.94.9 2.52.5 -0.4-0.4


LowLow 2020 2.82.8 4.64.6 1.71.7 --1.21.2

Hip FractureHip Fracture HighHigh 2121 3.23.2 8.18.1 4.94.9 +0.5+0.5


LowLow 1313 2.12.1 4.14.1 2.02.0 -2.4-2.4

CarotidCarotid

Endarter-Endarter-ectomyectomy

HighHigh 55 .32.32 .63.63 .31.31 -0.2-0.2

ModerateModerate 164164 .47.47 .98.98 .51.51

LowLow 66 .39.39 .39.39 .00.00 -0.51-0.51

What Impact Does Hospital Transfer Intensity Have on 30-day What Impact Does Hospital Transfer Intensity Have on 30-day Crude Mortality Rates Compared to Crude Inpatient Rates? Crude Mortality Rates Compared to Crude Inpatient Rates?

* Impact = Difference for High or Low group minus difference for moderate group* Impact = Difference for High or Low group minus difference for moderate group

Does Inclusion of POA Coding Improve the IMIs Does Inclusion of POA Coding Improve the IMIs Relative to a “Gold Standard”?Relative to a “Gold Standard”?

• Generated 2006 risk-adjusted mortality rates (RAMRs) Generated 2006 risk-adjusted mortality rates (RAMRs) for hospitals using CABG gold standard proxy for hospitals using CABG gold standard proxy

(Parker et al., Med Care 2006)(Parker et al., Med Care 2006)

• For same patient cohort, generated RAMRs using AHRQ For same patient cohort, generated RAMRs using AHRQ software with and without POAsoftware with and without POA

• Used inpatient mortality for both outcomesUsed inpatient mortality for both outcomes• Calculated correlation between AHRQ CABG IMI (with & Calculated correlation between AHRQ CABG IMI (with &

without POA) with CABG gold standard proxywithout POA) with CABG gold standard proxy• Correlation with gold standard proxy improved from .87 Correlation with gold standard proxy improved from .87

(without POA) to .93 (with POA): a 7% improvement(without POA) to .93 (with POA): a 7% improvement

Summary of FindingsSummary of Findings

• Some patient illness severity is not explained by APR-DRG risk Some patient illness severity is not explained by APR-DRG risk model (as demonstrated by DNR, PC & SNF status) but this does model (as demonstrated by DNR, PC & SNF status) but this does not strongly bias results (stroke a possible exception)not strongly bias results (stroke a possible exception)

• Hospitals that do a poor job of POA coding do not appear to Hospitals that do a poor job of POA coding do not appear to benefit in terms of their IMI resultsbenefit in terms of their IMI results

• Public hospitals had higher mortality rates for 5 out 6 IMIs: Both Public hospitals had higher mortality rates for 5 out 6 IMIs: Both Teaching and Investor hospitals had lowest rates for 3 IMIs. Teaching and Investor hospitals had lowest rates for 3 IMIs.

• Hospitals that rarely transfer patients to SNFs & other care Hospitals that rarely transfer patients to SNFs & other care perform more poorly when using inpatient mortality compared to perform more poorly when using inpatient mortality compared to 30-day mortality30-day mortality

– Transfer to SNF/other care rates by hospital Type: Transfer to SNF/other care rates by hospital Type: Non-profit = 15%, Investor = 20%, Public= 15%, Teaching = 10%Non-profit = 15%, Investor = 20%, Public= 15%, Teaching = 10%

• Utilization of POA coding improves assessment of hospital quality Utilization of POA coding improves assessment of hospital quality relative to a CABG gold standard proxyrelative to a CABG gold standard proxy

BACKUP SLIDESBACKUP SLIDES

Documents

Evaluating the AHRQ Inpatient Mortality Indicators for Public Reporting in California AHRQ Users Meeting Joseph Parker, PhD Brian Paciotti, PhD Merry Holliday-Hanson,