Profiling Hospitals by Survival of Patients with Colorectal Cancer

RESEARCH ARTICLE

Profiling Hospitals by Survival ofPatients with Colorectal CancerHui Zheng, Wei Zhang, John Z. Ayanian, Lawrence B. Zaborski,and Alan M. Zaslavsky

Objective. To profile hospitals by survival rates of colorectal cancer patients in mul-tiple periods after initial treatment.Data Sources. California Cancer Registry data from 50,544 patients receivingprimary surgery with curative intent for stage I–III colorectal cancer in 1994–1998,supplemented with hospital discharge abstracts.Study Design. We estimated a single Bayesian hierarchical model to quantify asso-ciations of survival to 30 days, 30 days to 1 year, and 1–5 years by hospital, adjusted forpatient age, sex, race, stage, tumor site, and comorbidities. We compared two profilingmethods for 30-day survival and four longer-term profiling methods by the fractions ofhospitals with demonstrably superior survival profiles and of hospital pairs whose rel-ative standings could be established confidently.Principal Findings. Interperiod correlation coefficients of the random effects arer̂12 ¼ 0:62 (95 percent credible interval 0.27, 0.85), r̂13 ¼ 0:52 (0.20, 0.76), and r̂23 ¼0:57 (0.19, 0.82). The three-period model ranks 5.4 percent of pairwise comparisons by30-day survival with at least 95 percent confidence, versus 3.3 percent of pairs using asingle-period model, and 15–20 percent by weighted multiperiod methods.Conclusions. The quality of care for colorectal cancer provided by a hospital system issomewhat consistent across the immediate postoperative and long-term follow-upperiods. Combining mortality profiles across longer periods may improve the statisticalreliability of outcome comparisons.

Key Words. Cancer care, colorectal cancer, provider profiling, quality measure-ment, Bayesian inference

Survival after initial treatment is an important indicator of the quality of cancercare. Short-term (30-day) postoperative survival is commonly regarded as themost proximate measure of the skill of the surgeon and quality of the hospitalin which surgery is performed (Schrag et al. 2000). Long-term survival is alsoimportant in assessing outcomes of care, since most deaths from colorectalcancer occur more than 30 days after surgery. Long-term survival may reflectthe provision of appropriate adjuvant therapy (National Institutes of Health

r Health Research and Educational TrustDOI: 10.1111/j.1475-6773.2010.01222.x

729

Health Services Research

1990; Andre et al. 2004), surveillance for cancer recurrence and new primarycancers (Renehan et al. 2002; Desch et al. 2005), and care of comorbid con-ditions. Currently, 5-year survival of patients with colorectal cancer in theUnited States exceeds 60 percent (American Cancer Society 2008), and morethan 1.1 million people who have been diagnosed with colorectal cancer werealive in 2004 (National Cancer Institute 2007).

Provider profiling allows for assessment and comparisons of the out-comes of health care providers (McNeil, Pedersen, and Gatsonis 1992; Epstein1995; Normand, Glickman, and Gatsonis 1997; Rosenthal et al. 1998). Resultsfrom profiling can influence policy making, referrals, and patients’ selection ofhealth care providers, and help with identifying and correcting quality prob-lems, although evidence on its impact is mixed (Mukamel and Mushlin 1998;Marshall et al. 2000; Hannan et al. 2003).

Profiling of surgical outcomes typically focuses on in-hospital or 30-daysurvival; little is known about the correlations between short- and long-termsurvival (Gatsonis et al. 1993; Normand, Glickman, and Gatsonis 1997). Ifsome provider system characteristics influence both short- and long-termoutcomes, we might expect positive correlations between short- and long-termperformance measures. Models incorporating such correlations among sur-vival rates in different periods may improve the statistical precision of profilingby combining information across periods. Furthermore, interperiod correla-tions of outcomes quantify the commonality of dimensions of quality per-taining to different phases of treatment. Therefore, such models are of bothclinical and statistical interest.

We focus on the hospital as the unit of analysis because cancer careacross the continuum from initial treatment to remission or death cannotgenerally be attributed to any single physician. Instead, care is typically de-livered by a cluster of physicians and facilities affiliated with a single hospital,which might reasonably be held accountable for the quality of the entireprocess of care (Fisher et al. 2009). Furthermore, hospitals are more likely thanphysicians to have sufficient numbers of patients to make profiling feasiblewith acceptable precision.

Address correspondence to Alan M. Zaslavsky, Ph.D., Department of Health Care Policy, Har-vard Medical School, 180 Longwood Avenue, Boston, MA 02115; e-mail: [email protected]. Hui Zheng, Ph.D., is with the Department of Medicine, Massachusetts General Hospital,Boston, MA. Wei Zhang, M.D., Ph.D., is with the China Europe International Business School,Beijing, China. John Z. Ayanian, M.D., M.P.P., and Lawrence B. Zaborski, M.S., are with theDepartment of Health Care Policy, Harvard Medical School, Boston, MA.

730 HSR: Health Services Research 46:3 ( June 2011)

mailto:[email protected]





METHODS

Study Cohort and Data Source

This study used data from the California Cancer Registry (CCR) on Californiaresidents diagnosed with stages I–III colorectal cancer between 1994 and 1998and receiving surgery with curative intent. Hospitals with fewer than 100patients (89 of 289 hospitals with 10.7 percent of all patients) were excludedfrom the study because they had too few patients to contribute reliable in-formation to the analysis.

The CCR data included patients’ age, gender, race/ethnicity, date ofdiagnosis, date of surgery, tumor site and stage, residence, and vital status. Toascertain patients’ vital status, registry staff matched records with the Califor-nia Death Statistical Master File and National Death Index. Socioeconomicstatus (SES) was represented by the median household income of each pa-tient’s census block of residence (rounded to the nearest U.S.$5,000 to pre-serve confidentiality), based on linked 1990 Census data.

CCR data were linked to hospital discharge abstracts maintained by theCalifornia Office of Statewide Health Planning and Development. We iden-tified patients’ discharge diagnosis codes shown on their hospital abstracts from18 months before to 6 months after their diagnosis with colorectal cancer. Toquantify the severity of comorbid illness, we calculated the Deyo modification ofthe Charlson comorbidity index for use with administrative data (Charlson et al.1987; Deyo, Cherkin, and Ciol 1992), excluding cancer diagnoses.

The institutional review boards of Harvard Medical School, theCalifornia Department of Health Services, and the Public Health Instituteapproved the study protocol. Because our study used existing data withencrypted identifiers, written informed consent was not required.

Study Variables

We adjusted for hospital case mix using the following patient characteristics asspecified in Table 1, including clinical (tumor site and stage in six combina-tions, comorbidity), demographic (age, gender, race/ethnicity), and socioeco-nomic (median household income) variables. Hospital volume was defined bythe number of patients undergoing cancer-related colorectal surgery in eachhospital, as recorded in the CCR from 1994 through 1998, and categorizedinto the three volume strata, low (100–150 patients, 57 hospitals), medium(151–300 patients, 80 hospitals), and high (4300 patients, 63 hospitals). Foreach physician reported to be involved in treating at least five patients in ourcohort, we tabulated the percentage of the physician’s patients seen in a single

Profiling Hospitals by Survival of Patients with Colorectal Cancer 731

hospital. The outcome measures were survival in three time periods: the first30 days after surgery, from 31 days to 1 year (among those surviving 30 days),and from 1 to 5 years (among those surviving 1 year).

Table 1: Cohort Characteristics and Coefficients of Individual and HospitalCharacteristics as Predictors of Mortality in Three Postsurgical Periods

% of Cohort

Estimated Coefficient (Standard Error) by Period

0–30 Days 30 Days to 1 Year 1–5 Years

Intercept � 4.044 (0.101) � 2.953 (0.057) � 1.071 (0.043)Age 18–54 12.3 � 1.565 (0.199) � 1.115 (0.084) � 0.811 (0.043)Age 55–59 7.1 � 0.924 (0.181) � 1.014 (0.100) � 0.817 (0.051)Age 60–64 9.4 � 1.041 (0.151) � 0.657 (0.076) � 0.638 (0.047)Age 65–69 13.4 � 0.771 (0.117) � 0.540 (0.066) � 0.512 (0.040)Age 70–74 16.6 � 0.515 (0.098) � 0.289 (0.054) � 0.308 (0.035)Age 75–79 17.2 Ref Ref RefAge 80–84 13.2 0.322 (0.082) 0.308 (0.051) 0.372 (0.038)Age 851 10.8 0.923 (0.077) 0.845 (0.052) 0.927 (0.041)Colon 1 17.5 � 0.497 (0.087) � 0.475 (0.058) � 0.522 (0.036)Colon 2 32.9 Ref Ref RefColon 3 24.7 0.052 (0.065) 0.798 (0.039) 0.779 (0.030)Rectal 1 7.9 � 0.456 (0.129) � 0.440 (0.086) � 0.371 (0.048)Rectal 2 8.4 � 0.116 (0.106) 0.184 (0.067) 0.370 (0.042)Rectal 3 8.7 � 0.118 (0.114) 0.654 (0.061) 1.037 (0.040)Income U.S. � $27,500 22.3 0.259 (0.082) 0.163 (0.051) 0.154 (0.034)Income U.S.$27,500–37,500 22.4 0.241 (0.081) 0.064 (0.052) 0.119 (0.033)Income U.S.$37,500–52,500 27.3 0.157 (0.078) 0.071 (0.049) 0.088 (0.032)Income U.S. � $52,500 23.3 Ref Ref RefIncome missingn 4.7 0.166 (0.138) 0.068 (0.085) � 0.072 (0.058)Female 49.5 � 0.196 (0.055) � 0.047 (0.033) � 0.229 (0.023)Male 50.5 Ref Ref RefWhite 75.7 Ref Ref RefAfrican American 6.0 0.018 (0.127) 0.001 (0.074) 0.217 (0.049)Hispanic 9.6 � 0.125 (0.102) � 0.087 (0.060) 0.009 (0.038)Asian 8.3 � 0.239 (0.128) � 0.359 (0.076) � 0.194 (0.045)Other race 0.4 0.315 (0.366) � 0.184 (0.276) 0.083 (0.163)Charlson 0 64.1 Ref Ref RefCharlson 1 21.9 0.756 (0.070) 0.503 (0.040) 0.373 (0.027)Charlson 2 7.8 1.358 (0.081) 0.979 (0.051) 0.810 (0.042)Charlson � 3 6.1 1.756 (0.077) 1.598 (0.050) 1.329 (0.048)Volume 100–150 13.7 0.202 (0.089) 0.233 (0.052) 0.076 (0.041)Volume 151–300 34.4 0.186 (0.072) 0.191 (0.040) 0.034 (0.033)Volume4300 51.9 Ref Ref Ref

Notes. Coefficients are from random effects logistic regression models.

Boldface represents po.05.nIncome missing due to address that could not be geocoded to block group.


Statistical Analysis

We fitted a logistic regression model for survival to the end of each period forpatients who were alive at the beginning of that period, with hospital randomeffects for each period, correlated across periods. Covariates included patientdemographic, socioeconomic and clinical characteristics, and hospital volumecategories, modeled with separate regression coefficients in each period. Weused Bayesian methods to fit the model with MLwiN software (Browne 2009).Details of model specification and fitting are presented in the Appendix SA2.

Model results included estimates of the variation of random effects foreach period across hospitals as well as the correlation of the effects betweeneach pair of periods, with 95 percent credible intervals (Bayesian equivalent ofconfidence intervals) for each estimate. For comparison we also fitted a similarmodel to only the period 1 data with a single random effect per hospital. Weestimated a principal components analysis of the correlation matrix to quan-tify the common variation of random effects across the three periods.

We obtained draws from the distribution of the random effect for eachhospital in each period. We combined these with the intercept and the co-efficient corresponding to the hospital’s volume category to obtain a ‘‘hospitalprofile,’’ defined as the combination of hospital-specific coefficients that couldbe used to compare predicted probabilities of mortality for identical patients.We refer to calculation of profiles for 30-day survival from the three-periodmodel as the ‘‘reference method.’’

As a summary of the precision of comparisons under various methods ofprofiling, we calculated posterior probabilities for each pair of hospitals in-dicating our degree of certainty that one hospital’s adjusted survival rate wassuperior to that of the other hospital. We calculated these comparisons usingprofiles from the base method, from the single-period model, and also usingcomposite profiles that combined profiles across periods of the three-periodmodel with each of four weightings: (1) equal weights; (2) z-score weights,where each period’s score is divided by the random-effects standard deviation(SD) st; (3) z-score weights, where each period’s score is divided by the em-pirical SD of the estimated profiles in each period; (4) weighted so the averageapproximates the probability of survival across the three periods (AppendixSA2).

We also calculated the posterior probability that each hospital wasamong the 25 percent of hospitals with the best survival profiles. Finally, wecalculated correlations across hospitals of the profiles estimated using these sixmethods.


RESULTS

Characteristics of Study Cohort

The CCR data for the study period included 56,539 eligible patients at 289hospitals. After exclusion of hospitals with fewer than 100 patients, our studycohort had 50,544 eligible patients (89.3 percent of all patients) at 200 hos-pitals. Of these, 49.5 percent were female, 75.7 percent white, 6.0 percentAfrican American, 8.3 percent Asian, and 9.6 percent Hispanic, reflecting theracially diverse population of our site. The mean age of the cohort was 70.2years (SD 12.6).

At the 200 hospitals included in the study, the patient-weighted mediannumber of patients undergoing surgery was 307 cases with an interquartilerange of 200–423 cases. The unadjusted 30-day, 31 day to 1 year, and 1–5years survival rates were 97.0, 91.2, and 69.8 percent, respectively. Amongphysicians seeing at least five patients in our cohort, those for whom at least 80percent of their patients received primary treatment in a single hospitalincluded between 70 and 74 percent among attending physicians, medicaloncologists, radiation oncologists, and surgeons.

Effects of Patient and Hospital Characteristics on Survival

Table 1 shows regression coefficients of patient and hospital characteristics ineach time period. Increasing age, low income, and greater comorbidity weresignificant predictors of mortality in each period, while mortality was signifi-cantly lower in each period for stage I colon or rectal cancer patients com-pared with stage II colon patients. Other significant predictors of highermortality in some but not all periods included treatment in a low- or medium-volume hospital, African American race, male sex, and (after 30 days) stage IIIcolon cancer and stage II or III rectal cancer. Asian race predicted lowermortality after 30 days.

The estimated correlation coefficients rst of the random effects across thethree time periods were r12 5 0.62 (95 percent credible interval 0.27, 0.85),r13 5 0.52 (CI 5 0.20, 0.76), and r23 5 0.57 (CI 5 0.19, 0.82). These positivecorrelations indicate that hospitals that have better survival in one period tendto have better survival in other periods as well. Principal components analysisof the correlation matrix showed that 70 percent of the variance in the threecorrelated random effects is explained by a single principal component.

Estimated SDs of the hospital random effects (on the logit scale) ineach period were s1 5 0.25 (CI 5 0.17, 0.31), s2 5 0.09 (CI 5 0.02, 0.13),and s3 5 0.14 (CI 5 0.10, 0.16). These SDs can be interpreted as measures of


relative variation across hospitals of hazard of mortality, which thus is largestin the 30-day postoperative period. To illustrate, for a patient with average riskof 30-day mortality (3.0 percent) at a hospital with median mortality rate, therisks of mortality, respectively, at hospitals in the same volume category withmoderately low (1 SD below median on the logit scale) and moderately high(1 SD above) mortality would be 2.4 and 3.8 percent. Corresponding mod-erately low and high risks of 30-day to 1-year mortality would be 8.1 and 9.5percent, and for 1–5-year mortality would be 27.3 and 33.2 percent. Therandom effect SD was larger than the effect of hospital volume in periods 1 and3, and of comparable magnitude in period 2.

Hospital Profiles for 30-Day Survival

Figure 1 illustrates hospital profiles for 30-day survival estimated under thereference method (using information from all three periods), represented bythe linear prediction for a patient with average characteristics. Hospitals aredisplayed in the order of the posterior median case mix-adjusted survival rate.The 90 and 95 percent credible intervals were wide compared with the vari-ation in posterior medians, and only two hospitals were shown to be above orbelow the overall median with probability 495 percent. The correspondingfigure for intervals calculated under the single-period model was similar (datanot shown).

500 100 150 200

3.0

4.0

5.0

Profile

Hospital

Figure 1: Profiling Results for 30-Day Survival Using the Reference Method,with Hospitals Ordered by Estimated Profile for Survival (Logit of Probabilityof Survival for Patient with Mean Covariate Values)

The horizontal bar represents the median of the posterior distribution of provider profile. Each

line segment represents the 95 percent posterior interval, with short horizontal line segments for

the limits of the 90 percent posterior interval.


Pairwise Comparisons and High-Ranking Hospitals Using Multiple Profiling Methods

We assessed the discriminating power of each of the six profiling methodsdescribed above by the number of pairwise comparisons between hospitalprofiles that have a ‘‘significant’’ posterior comparison probability (40.95) outof 19,900 possible pairwise comparisons (Table 2). The reference methodyielded significant comparisons for 1,076 (5.4 percent) of such pairs, while thesingle-period method did so for only 3.3 percent of pairs. The methods thatweighted together profiles from the three periods distinguish many more pairs,ranging from 15.1 percent (Method 4) to 21.2 percent (Method 2). Similarrelative results were obtained when the criterion was reduced to 90 percentcertainty, with 31.1 percent of pairs being ranked at that level by Method 2.

Figure 2 illustrates pairwise comparisons for the reference method(above diagonal) and weighted Method 2 (below diagonal), where blacksquares represent hospital pairs that can be ordered with probability 495percent and dark and light gray squares, respectively, represent pairs with 90percentoprobabilityo95 percent and 80 percentoprobabilityo90 percent.Hospitals whose profile scores differed more (further from the diagonal in thefigure) were more likely to be confidently ordered. The greater density of theplot below the diagonal graphically illustrates the greater precision of com-parisons using Method 2.

We compared methods on their ability to identify high-performing hos-pitals, defined as those with profiles in the top 25 percent, with at least 80percent certainty (Table 2). While only eight hospitals could be so identifiedby the reference method (Figure 3), 24 (or almost half of the 50 in the top 25percent) could be so identified using Method 2.

Consistency among Profiling Methods

Correlations among profiles from the six profiling methods were high (Table 2).The correlation of reference-method profiles with single-period profiles is 0.944and with the weighted averages ranged from 0.880 to 0.962. Correlations be-tween weighted average methods ranged from 0.967 to 0.996; among thesemethods, Method 4 yields the lowest correlations with the other methods.

DISCUSSION

One of the key challenges of health care provider profiling is that ‘‘gold-standard’’ outcome measures, such as mortality in specific episodes of illness,


Tab

le2:

Com

par

ison

sam

ong

Six

Pro

filin

gM

eth

ods:

Dis

crim

inat

ing

Ab

ility

and

Cor

rela

tion

s

Pro

filin

gM

etho

d

Num

ber

(%)

ofSi

gnifi

cant

Pai

rwis

eH

ospi

talC

ompa

riso

nsn

Num

ber

ofH

ospi

tals

wit

hp4

80%

ofB

eing

inB

est

25%

Cor

rela

tion

sw

ith

Pro

files

from

Oth

erM

etho

ds

At4

95%

Pro

babi

lity

Lev

el

At4

90%

Pro

babi

lity

Lev

elP

erio

d1,

Sing

le-

Per

iod

Mod

el

Met

hod

1A

vera

geP

rofil

e

Met

hod

2A

vera

geP

rofil

e

Met

hod

3A

vera

geP

rofil

e

Met

hod

4A

vera

geP

rofil

e

Per

iod

1p

rofil

e,3-

per

iod

mod

el(‘‘

refe

ren

cem

eth

od’’)

1,07

6(5

.4%

)2,

613

(13.

1%)

80.

940.

960.

920.

940.

88

Per

iod

1p

rofil

e,si

ngl

e-p

erio

dm

odel

676

(3.3

%)

1,84

0(9

.2%

)3

10.

850.

810.

810.

71

Met

hod

1av

erag

ep

rofil

ew3,

130

(15.

7%)

5,14

6(2

5.9%

)18

10.

994

0.99

0.97

Met

hod

2av

erag

ep

rofil

ew4,

215

(21.

2%)

6,19

2(3

1.1%

)24

10.

990.

97M

eth

od3

aver

age

pro

filew

3,51

4(1

7.7%

)5,

536

(27.

8%)

201

0.98

Met

hod

4av

erag

ep

rofil

ew2,

995

(15.

1%)

4,93

1(2

4.8%

)17

1

nP

erce

nta

ges

are

rela

tive

to19

,900

pos

sib

lep

airw

ise

com

par

ison

sof

200

hos

pit

als.

w Ave

rage

pro

files

wei

ghti

ngs

are

(1)e

qual

wei

ghts

;(2)

z-sc

ored

by

ran

dom

-eff

ects

stan

dar

dd

evia

tions t

;(3)

z-sc

ored

by

emp

iric

alst

and

ard

dev

iati

onof

the

esti

mat

edp

rofil

esin

each

per

iod

;(4

)w

eigh

ted

toap

pro

xim

ate

pro

filin

gb

yth

ep

rob

abili

tyof

surv

ival

thro

ugh

the

thre

ep

erio

ds.


may have poor statistical reliability simply because the number of deaths orother adverse outcomes for any provider is too small. Strategies for addressingthis challenge include combining information across outcome measures orconditions, using process measures or intermediate clinical outcomes such ascomplication rates, pooling information over a longer time period, or defininga larger unit of assessment (e.g., a hospital instead of an individual physician)(Krumholz et al. 2002; Jha 2006; Clancy 2007; Jha et al. 2007; Kerr et al. 2007).Each of these strategies involves a compromise in which greater statisticalpower is achieved at the cost of shifting the focus from what might otherwisebe regarded as the target of ‘‘gold-standard’’ measurement, typically a direct

Figure 2: Significant Pairwise Comparisons of Hospital Profiles

Triangle above diagonal is for reference method, below diagonal is for weighted Method 2. In

each triangle, hospitals are ordered by posterior median profile. Black squares indicate pairs for

which ordering of profiles is 495 percent certain, dark gray squares indicate pairs for which

ordering is between 90 and 95 percent certain, and light gray squares indicate pairs for which

ordering is between 80 and 90 percent certain. The white area adjacent to diagonal represents pairs

whose profiles are not sufficiently different to be ordered with probability 480 percent.


estimate of a key clinical outcome (such as survival) during a recent timeperiod for a single condition.

In this study, we combined two of these strategies (an extended 5-yearsampling window and aggregation of data to the hospital level) with a third,extending the follow-up window for assessment of hospital-specific mortalityrisk. While 30-day postsurgical mortality is a common measure of the qualityof surgery and postoperative care for colorectal cancer, only 7.9 percent of 5-year mortality for our cohort occurred during the first 30 days, limiting theamount of information available for comparisons among hospitals. Conse-quently only 3.2 percent of pairs of hospitals could be comparatively rankedwith 95 percent certainty, using these data in combination with hospital vol-ume. Because surgical volume accounts for only a modest part of the variationin mortality rates, it is essential to seek other information that could improvethe precision of comparisons among hospitals.

Hence, we investigated associations of 30-day mortality rates by hospitalwith mortality among survivors of this initial period for the same hospitals,divided into mortality from 31 days to 1 year postoperatively, and survivalfrom 1 to 5 years. During the second of these three periods, adjuvant therapieswould typically be provided if clinically indicated, whereas cancer surveil-lance and survivorship care are the focus of the third period representinglonger-term survival to 5 years. A multiperiod model suggested strong cor-relations of survival rates among the three time periods after adjustment forpatient case mix and hospital volume. Clinically, these associations suggestthat there might be a common dimension of hospital quality affecting theentire 5-year period. This might be attributable in part to persistent effects of

0 50 100 150 200

0.0

0.4

0.8

Proba

bility

Hospital

Figure 3: Posterior Probabilities That Each Hospital’s Profile Is in the Top 25Percent of All Hospitals

The eight hospitals crossing the horizontal line have probabilities 480 percent of being in the

top 25 percent.


high-quality initial treatment and in part to a generalized dimension of qualitymanifested in better follow-up care by providers associated with higher-qualityhospitals. Effects of hospital volume on mortality were similar across periods;other hospital characteristics not available for our analysis, such as processmeasures of follow-up care and surveillance, and structural measures of in-tegration of primary and specialty care, might help to explain some of theresidual common variation in quality (Bradley et al. 2006).

Statistically, these associations suggest that profiling could be made moreprecise by incorporating information from the later postoperative periods.This could be approached in two conceptually distinct ways. One approachwould maintain 30-day mortality as the primary outcome measure and regardlater mortality rates only as auxiliary information to improve estimation of this30-day measure. This method, implemented as the ‘‘reference method’’ in ouranalysis, modestly improves on the model that uses only 30-day mortalitydata, almost doubling the number of hospital comparisons that can be madewith 95 percent confidence.

Alternately, the strong positive correlations of hospital mortality ratesacross postoperative periods suggest that an index combining informationacross periods might be appropriate for hospital profiling. Such indices wouldno longer be interpreted purely as a measure of the quality of surgical care, butrather as a summary of the quality of care over a more extended period byproviders serving patients with colorectal cancer of that hospital. Becauseinformation across all three periods is used more directly rather than only as apredictor of mortality rates in the first period, this alternative objective allowsfor more powerful comparisons among hospitals.

The four weighting methods for combining profiles across periods allgenerate more significant pairwise comparisons than the methods based on asingle period profile. Different relative weightings of the periods lead to fairlysimilar results. A method that approximates profiling by total 5-year mortalityproduced results most dissimilar to those from 30-day mortality, because it wasmost affected by later (1–5-year postoperative) mortality, which constitutes themajority of total mortality in the first 5 years. The most statistically reliableprofiles were produced by normalizing the profiles in each period to a com-mon scale using z scores before combining them (Method 2). Selection of amethod for synthesizing survival across the three periods should consider boththe statistical precision of the estimates (which favors Method 2) and the policyobjective of giving substantial weight to direct measures of performance forprimary treatment in the hospital, where relative variation is greatest (whichfavors the highly correlated Methods 1–3).


Relative variation across hospitals in mortality is greatest in the imme-diate postoperative period. This is not surprising because the hospital’s med-ical staff and facilities are likely to have the strongest effect on patient careduring and immediately after inpatient treatment, while services involved inoutpatient follow-up involve a more diffuse network of providers. Nonethe-less, we did find significant variation up to 5 years postoperatively, reflectingsome combination of persistent effects of the quality of initial treatment anddifferences in the quality of follow-up care in the networks of physicians affil-iated with different hospitals. None of the methods could reliably distinguishthe better hospital in the majority of possible pairs, which is understandablebecause most hospitals fall into an intermediate range of similar quality.Nonetheless, hospitals that are fairly far apart in quality are likely to be dis-tinguishable with 90 percent certainty. While this falls below the usual cri-terion of significance for generalizable scientific findings, we believe that aninformed consumer or referring physician would reasonably consider thisevidence when choosing a treatment facility (Davis, Hibbard, and Milstein2007). Similarly high or low performers can be identified as potential sites forstudying best practices or problems, respectively, with adequate confidencefor targeting quality improvement efforts.

Previous analyses of the same cohort showed variation across hospitalsin other measures of treatment quality for colorectal cancer, such as rates ofadjuvant chemotherapy for eligible patients and permanent colostomy ratesafter rectal cancer surgery (Hodgson et al. 2003; Rogers et al. 2006; Zhenget al. 2006; Zhang, Ayanian, and Zaslavsky 2007). Future studies mightconsider whether it would be appropriate to combine such measures withprofiles of survival in a more comprehensive quality index. Other possibleextensions involve combining information across conditions (other cancersites), using similar statistical methods to determine which measures are mostcorrelated and therefore might most appropriately be combined in profiling.

Our study had several limitations. First, we had only limited measures ofpatient characteristics, and our SES measures were limited to area-based in-come estimates. If patients with higher individual SES receive initial treatmentat hospitals of higher quality and then have better resources and social supportover the following years, correlations over time might be observed that are dueto this unmeasured variation in patient case mix rather than hospital quality.Additional information that might be useful would include individual-levelmeasures of income, education, and psychosocial support; more detailedclinical measures, including severity of comorbidity, functional status, andbody mass index; and patient reports on interactions with the health care


system. Such variables might help to explain the superior survival of Asianpatients reported here and in previous studies (Hodgson et al. 2003; Edwardset al. 2005; Rogers et al. 2006).

Second, our models included only a single hospital structural measure(treatment volume) and no process measures. Additional measures of relevanthospital characteristics should improve the precision of profiling and help toidentify the drivers of quality across periods. Third, we did not have reliabledata on cause of death and therefore could not distinguish cancer-related andunrelated deaths. Fourth, we relied on hospital coding of comorbid conditions,which might not have been uniform, especially given the different incentivesto hospitals in fee-for-service versus closed health care systems (Quan, Par-sons, and Ghali 2002). Fifth, our data were limited to one state; future studiesmight investigate the consistency of these findings across more states withmore recent data. We omitted the smallest 31 percent of hospitals, accountingfor 10.7 percent of patients, from our analyses; these hospitals had too fewpatients to support reliable mortality profiling. Sixth, we could not determinewhether the groupings of providers involved in treatment of our hospital-centered patient clusters correspond to physician networks that might beformed under an accountable care organization (ACO) payment model, al-though we did find that most physicians in any role tended to treat patientsmost of whom had received primary therapy at a single hospital. Seventh, forsimplicity of presentation, volume was coded using indicator variables withsomewhat arbitrary cutpoints; more flexible specifications might be exploredin a profiling application. Finally, even with a dataset representing relativelylarge numbers of patients and hospitals, substantial statistical uncertainty re-mains in estimation of the random effects covariance parameters. While ouranalyses properly reflect this uncertainty, accumulation of more data overtime and over more extensive areas would enable more precise estimation ofthese parameters and consequently sharpen inferences for model-based pro-filing. Such regional- or national-level modeling would also be necessary toapply these methods in smaller states where independent estimates of modelparameters would be less precise.

In the future, quality measurement is likely to focus on assessing out-comes of care provided by ACOs over entire episodes of illness, integratingquality in inpatient and outpatient settings, rather than on discrete processes orphases of care (Fisher et al. 2009). Such ACOs, many of which are likely to behospital-centered, are authorized under the recently enacted Patient Protec-tion and Accountable Care Act (Merlis 2010). Initiatives such as the RapidQuality Reporting System of the American College of Surgeons are using this


approach to evaluate the quality of cancer care (American College of Surgeons2009). Methods such as those we have demonstrated to integrate outcomesover sequential time periods may contribute to developing and validating suchquality measures.

ACKNOWLEDGMENTS

Joint Acknowledgment/Disclosure Statement: This study was supported by grantsR01 HS09869 and U01 CA93324 from the Agency for Healthcare Researchand Quality and the National Cancer Institute. The authors are grateful toWilliam E. Wright, Ph.D., and Mark E. Allen, M.S., for facilitating access toCalifornia Cancer Registry data; to Craig Earle, M.D., for insightful com-ments; and to Robert Wolf, M.S., for data preparation.

The collection of cancer data used in this study was supported by theCalifornia Department of Health Services through the statewide cancer re-porting program mandated by California Health and Safety Code Section103885, the National Cancer Institute’s SEER Program, and the Centers forDisease Control and Prevention’s National Program of Cancer Registries. Theideas and opinions expressed herein are those of the authors, and endorse-ment by the State of California Department of Health Services, the NationalCancer Institute, and the Centers for Disease Control and Prevention is notintended nor should be inferred.

Disclosures: None.Disclaimers: None.

REFERENCES

American Cancer Society. 2008. Cancer Facts & Figures 2008. Atlanta: American CancerSociety.

American College of Surgeons. 2009. ‘‘Rapid Quality Reporting System’’ [accessed onFebruary 4, 2009]. Available at http://www.facs.org/cancer/ncdb/rqrs0509.pdf

Andre, T., C. Boni, L. Mounedji-Boudiaf, M. Navarro, J. Tabernero, T. Hickish, C. Top-ham, M. Zaninelli, P. Clingan, J. Bridgewater, I. Tabah-Fisch, and A. de Gramont.2004. ‘‘Oxaliplatin, Fluorouracil, and Leucovorin as Adjuvant Treatment for ColonCancer.’’ New England Journal of Medicine 350 (23): 2343–51.

Bradley, E. H., J. Herrin, B. Elbel, R. L. McNamara, D. J. Magid, B. K. Nallamothu, Y.Wang, S. L. Normand, J. A. Spertus, and H. M. Krumholz. 2006. ‘‘HospitalQuality for Acute Myocardial Infarction: Correlation among Process Measures


http://www.facs.org/cancer/ncdb/rqrs0509.pdf

and Relationship with Short-Term Mortality.’’ Journal of the American MedicalAssociation 296 (1): 72–8.

Browne, W. J. 2009. MCMC Estimation in MLwiN, v2.13. Bristol, England: Centre forMultilevel Modelling, University of Bristol.

Charlson, M. E., P. Pompei, K. L. Ales, and C. R. MacKenzie. 1987. ‘‘A New Method ofClassifying Prognostic Comorbidity in Longitudinal Studies: Development andValidation.’’ Journal of Chronic Diseases 40 (5): 373–83.

Clancy, C. 2007. ‘‘The Performance of Performance Measurement.’’ Health ServicesResearch 42 (5): 1797–801.

Davis, M., J. H. Hibbard, and A. Milstein. 2007. Consumer Tolerance for Inaccuracy inPhysician Performance Ratings: One Size Fits None. Washington, DC: Center forStudying Health Systems Change.

Desch, C. E., A. B. III Benson, M. R. Somerfield, P. J. Flynn, C. Krause, C. L. Loprinzi,B. D. Minsky, D. G. Pfister, K. S. Virgo, and N. J. Petrelli. 2005. ‘‘ColorectalCancer Surveillance: 2005 Update of an American Society of Clinical OncologyPractice Guideline.’’ Journal of Clinical Oncology 23 (33): 8512–9.

Deyo, R. A., D. C. Cherkin, and M. A. Ciol. 1992. ‘‘Adapting a Clinical ComorbidityIndex for Use with ICD-9-CM Administrative Databases.’’ Journal of ClinicalEpidemiology 45 (6): 613–9.

Edwards, B. K., M. L. Brown, P. A. Wingo, H. L. Howe, E. Ward, L. A. Ries, D. Schrag,P. M. Jamison, A. Jemal, X. C. Wu, C. Friedman, L. Harlan, J. Warren, R. N.Anderson, and L. W. Pickle. 2005. ‘‘Annual Report to the Nation on the Status ofCancer, 1975–2002, Featuring Population-Based Trends in Cancer Treatment.’’Journal of the National Cancer Institute 97 (19): 1407–27.

Epstein, A. M. 1995. ‘‘Performance Reports on Quality Prototypes, Problems, andProspects.’’ New England Journal of Medicine 333 (1): 57–61.

Fisher, E. S., M. B. McClellan, J. Bertko, S. M. Lieberman, J. J. Lee, J. L. Lewis, and J. S.Skinner. 2009. ‘‘Fostering Accountable Health Care: Moving Forward in Med-icare.’’ Health Affairs (Millwood) 28 (2): w219–31.

Gatsonis, C. A., S. L. Normand, C. Liu, and C. Morris. 1993. ‘‘Geographic Variation ofProcedure Utilization: A Hierarchical Model Approach.’’ Medical Care 31:YS54–9.

Hannan, E. L., M. S. Sarrazin, D. R. Doran, and G. E. Rosenthal. 2003. ‘‘ProviderProfiling and Quality Improvement Efforts in Coronary Artery Bypass GraftSurgery: The Effect on Short-Term Mortality among Medicare Beneficiaries.’’Medical Care 41 (10): 1164–72.

Hodgson, D. C., W. Zhang, A. M. Zaslavsky, C. S. Fuchs, W. E. Wright, and J. Z.Ayanian. 2003. ‘‘Relation of Hospital Volume to Colostomy Rates and Survivalfor Patients with Rectal Cancer.’’ Journal of the National Cancer Institute 95 (10):708–16.

Jha, A. K. 2006. ‘‘Measuring Hospital Quality: What Physicians Do? How PatientsFare? Or Both?’’ Journal of the American Medical Association 296 (1): 95–97.

Jha, A. K., E. J. Orav, Z. Li, and A. M. Epstein. 2007. ‘‘The Inverse Relationshipbetween Mortality Rates and Performance in the Hospital Quality AllianceMeasures.’’ Health Affairs (Millwood) 26 (4): 1104–10.


Kerr, E. A., T. P. Hofer, R. A. Hayward, J. L. Adams, M. M. Hogan, E. A. McGlynn,and S. M. Asch. 2007. ‘‘Quality by Any Other Name? A Comparison of ThreeProfiling Systems for Assessing Health Care Quality.’’ Health Services Research42 (5): 2070–87.

Krumholz, H. M., S. S. Rathore, J. Chen, Y. Wang, and M. J. Radford. 2002. ‘‘Eval-uation of a Consumer-Oriented Internet Health Care Report Card: The Risk ofQuality Ratings Based on Mortality Data.’’ Journal of the American MedicalAssociation 287 (10): 1277–87.

Marshall, M. N., P. G. Shekelle, S. Leatherman, and R. H. Brook. 2000. ‘‘ThePublic Release of Performance Data: What Do We Expect to Gain? AReview of the Evidence.’’ Journal of the American Medical Association 283 (14):1866–74.

McNeil, B., S. Pedersen, and C. Gatsonis. 1992. ‘‘Current Issues in Profiling Quality ofCare.’’ Inquiry 29: 298–307.

Merlis, M. 2010. ‘‘Health Policy Brief. Accountable Care Organizations’’ [accessed onAugust 13, 2010]. Available at http://www.healthaffairs.org/healthpolicybriefs/brief.php?brief_id=23

Mukamel, D. B., and A. I. Mushlin. 1998. ‘‘Quality of Care Information Makes aDifference: An Analysis of Market Share and Price Changes after Publication ofthe New York State Cardiac Surgery Mortality Reports.’’ Medical Care 36 (7):945–54.

National Cancer Institute. 2007. Cancer Trends Progress Report——2007 Update. Bethesda,MD: National Cancer Institute, NIH, DHHS.

National Institutes of Health. 1990. ‘‘NIH Consensus Conference: Adjuvant Therapyfor Patients with Colon and Rectal Cancer.’’ Journal of the American MedicalAssociation 264: 1444–50.

Normand, S. L., M. E. Glickman, and C. A. Gatsonis. 1997. ‘‘Statistical Methods forProfiling Providers of Medical Care: Issues and Applications.’’ Journal of theAmerican Statistical Association 92 (439): 803–14.

Quan, H., G. A. Parsons, and W. A. Ghali. 2002. ‘‘Validity of Information on ComorbidityDerived from ICD-9-CCM Administrative Data.’’ Medical Care 40 (8): 675–85.

Renehan, A. G., M. Egger, M. P. Saunders, and S. T. O’Dwyer. 2002. ‘‘Impact onSurvival of Intensive Follow Up after Curative Resection for Colorectal Cancer:Systematic Review and Meta-Analysis of Randomised Trials.’’ British MedicalJournal 324 (7341): 813.

Rogers, S. O. Jr., R. E. Wolf, A. M. Zaslavsky, W. E. Wright, and J. Z. Ayanian. 2006.‘‘Relation of Surgeon and Hospital Volume to Processes and Outcomes ofColorectal Cancer Surgery.’’ Annals of Surgery 244 (6): 1003–11.

Rosenthal, G. E., A. Shah, L. E. Way, and D. L. Harper. 1998. ‘‘Variations in Stan-dardized Hospital Mortality Rates for Six Common Medical Diagnoses: Impli-cations for Profiling Hospital Quality.’’ Medical Care 36: 955–64.

Schrag, D., L. D. Cramer, P. B. Bach, A. M. Cohen, J. L. Warren, and C. B. Begg.2000. ‘‘Influence of Hospital Procedure Volume on Outcomes FollowingSurgery for Colon Cancer.’’ Journal of the American Medical Association 284 (23):3028–35.


http://www.healthaffairs.org/healthpolicybriefs/brief.php?brief_id=23

http://www.healthaffairs.org/healthpolicybriefs/brief.php?brief_id=23

Zhang, W., J. Z. Ayanian, and A. M. Zaslavsky. 2007. ‘‘Patient Characteristics andHospital Quality for Colorectal Cancer Surgery.’’ International Journal for Qualityin Health Care 19 (1): 11–20.

Zheng, H., R. Yucel, J. Z. Ayanian, and A. M. Zaslavsky. 2006. ‘‘Profiling Providers onUse of Adjuvant Chemotherapy by Combining Cancer Registry and MedicalRecord Data.’’ Medical Care 44 (1): 1–7.

SUPPORTING INFORMATION

Additional supporting information may be found in the online version of thisarticle:

Appendix SA1: Author Matrix.Appendix SA2: Statistical Appendix.

Please note: Wiley-Blackwell is not responsible for the content or func-tionality of any supporting materials supplied by the authors. Any queries(other than missing material) should be directed to the corresponding authorfor the article.


Documents

Profiling Hospitals by Survival of Patients with Colorectal Cancer