Benchmarking, public reporting, and pay-for-performance: A mixed-methods survey of California pediatric intensive care unit medical directors

Benchmarking, public reporting, and pay-for-performance: Amixed-methods survey of California pediatric intensive care unitmedical directors

JoAnne E. Natale, MD, PhD; Jill G. Joseph, MD, PhD; Ryan D. Honomichl, PhD; Lianna G. Bazanni, PhD, MPH;Kimie J. Kagawa, MD; James P. Marcin, MD, MPH

T here has been increased inter-est in, and efforts directed to,measuring and improving thequality of health care delivered

in the United States, particularly expen-sive hospital-based and critical care med-icine (1). Among the drivers for these

escalating costs is the significant variabil-ity in care across intensive care units(ICUs) at different institutions (2). Effortsfocused on measuring and improvingquality are particularly vital when dis-cussing the care of critically ill and in-jured infants and children, not only be-cause of the increasing cost of intensivecare but also because inadequate qualityof care likely results in long-term mor-bidity (3).

Benchmarking, public reporting, andpay-for-performance are measures thatare a means of addressing quality andcosts of health care (4–6). Benchmarkingprovides participating institutions withinformation about their performance rel-ative to comparable institutions. Thismeasure has been proposed to help re-duce variability in care across institutionsand in communicating quality perfor-mance comparisons to consumers (7).Such information may be either providedconfidentially to individual institutionsor more widely available as public report-ing or “report cards.” Initiatives such as

public reporting are seen as ways of em-powering the public, healthcare purchas-ers, and clinical providers to make aninformed selection of healthcare services(8). For example, public reports of hospi-tal and physician performance and out-come data have been credited with stim-ulating quality improvement at thehospital and physician levels (4, 9–11).Pay-for-performance links provider orhospital reimbursements (or both) to de-fined outcomes by financially rewardinghigh-quality and efficient care (8).

However, there have been concernsabout the unintended and potentially ad-verse consequences of benchmarking,public reporting, and pay-for-perfor-mance. For example, hospitals, clinicalservices, or physicians might be moti-vated to either select low-risk patients oravoid high-risk patients to maintainhigh-quality rankings or more financialreward or both (12–16). In addition, thevalidity and the appropriate interpreta-tion of data are critically important tobenchmark, report to the public, and de-

From the Department of Pediatrics (JEN, RDH,LGB, JPM) and Center for Healthcare Policy and Re-search (JPM), University of California Davis, Davis, CA;Center for Clinical and Community Research (JGJ),Children’s National Medical Center, Washington, DC;and California Children’s Services (KJK), Department ofHealth Care Services, Sacramento, CA.

Supported, in part, by Department of Health CareServices, State of California.

Dr. Kagawa has received grant support from theCalifornia Department of Health Care Services. Dr.Bazanni receives funding from the California Depart-ment of Health Care Services. The remaining authorshave not disclosed any potential conflicts of interest.

For information regarding this article, E-mail:[email protected]

Copyright © 2011 by the Society of Critical CareMedicine and the World Federation of Pediatric Inten-sive and Critical Care Societies

DOI: 10.1097/PCC.0b013e3181fe2e26

Objectives: We sought to assess the attitudes of pediatricintensive care unit medical directors in California regarding theneed for, the validity of, and the potential impact of benchmark-ing, public reporting, and pay-for-performance on pediatric crit-ical care.

Design: Cross-sectional survey.Setting: Pediatric intensive care units in California.Subjects: Medical director of pediatric intensive care units.Interventions: None.Measurements and Main Results: Self-administered question-

naire and a semi-structured phone interview from 16 pediatricintensive care unit medical directors. All data were anonymizedbefore review. Standard methods for identifying and agreeing onthemes in transcribed interviews were applied. Seventy-threepercent of California pediatric intensive care unit medical direc-tors agree that benchmarking improves patient outcomes but areundecided whether public reporting and pay-for-performance im-prove healthcare quality. They are wary of the validity of data

used to generate these performance measures and are discour-aged by the time and costs required to collect data for standardperformance outcomes (severity-adjusted pediatric intensive careunit mortality). Leadership opinions appear potentially “dynamic”in multiple domains and across each of the measures assessed.

Conclusions: Pediatric intensive care unit medical directorssometimes express contradictory opinions about the merits ofshared benchmarking efforts and express concerns across arange of logistic, methodological, and policy issues. These find-ings raise fundamental questions about how to create clinicalperformance standards that facilitate quality improvement in theface of a seriously divided constituency. Further, we propose thatpediatric intensive care unit medical directors play more activeroles in the development, implementation, and communication ofshared state-wide data collection. (Pediatr Crit Care Med 2011;12:000–000)

KEY WORDS: benchmarking; public reporting; pay-for-perfor-mance; healthcare quality; pediatric critical care

1Pediatr Crit Care Med 2011 Vol. 12, No. 5

AQ:1

AQ: 8

AQ: 9AQ: 10

AQ: 2

balt5/zk8-pcc/zk8-pcc/zk800511/zk83766-10z xppws S�1 10/25/10 5:39 Art: 200751 Input-ht

termine pay-for-performance criteria (7,17). If clinicians believe that the perfor-mance and outcome data are not valid,then there may be reluctance to partici-pate in such initiatives and to use themfor motivating and informing practiceimprovement programs. Little is specifi-cally known about the beliefs and atti-tudes of pediatric intensive care unit(PICU) medical directors regarding thevalidity or value of PICU performance andoutcome data in impacting provider be-havior or public perceptions relevant toquality.

This issue is particularly salient in thenation’s largest state, California, wheresuch reports are being considered by Cal-ifornia Children’s Services (CCS), thestate office that defines PICU clinical ser-vice and quality standards. In July 2007,19 PICUs meeting CCS standards wereeligible for supplemental Medi-Cal (Cali-fornia state version of Medicaid) reim-bursement. In return, CCS requires thesePICUs to annually report the number ofPICU admissions, mortality, and risk-adjusted mortality using the PediatricRisk of Mortality III score. CCS has beenusing these data for internal purposesonly without any dissemination. How-ever, the same or more comprehensiveoutcome measures could be used to pop-ulate comparative reports that could bemore widely distributed or even publiclyavailable.

Because CCS is currently consideringsuch an application, we sought to assessthe attitudes of PICU medical directors inCalifornia regarding the need for, the va-lidity of, and potential impact that bench-marking, public reporting, and pay-for-performance might have on pediatriccritical care.

MATERIALS AND METHODS

Sample and Recruitment. Medical direc-tors from all 19 CCS-approved PICUs wereinvited to participate in a survey to determinetheir attitudes regarding state-mandated re-porting of their units’ outcomes.

Survey Methods. We developed a struc-tured survey instrument to assess the atti-tudes of these PICU medical directors regard-ing the need for statewide benchmarkingreports. Based on a literature review com-pleted in March 2007, a structured survey in-strument was developed by all authors. Thesurvey was pilot-tested among three indepen-dent academic PICU attendings at the Univer-sity of California Davis. The instrument wasthen revised and reviewed by the CCS programstaff, including one PICU physician and twogeneral pediatricians. The final instrument is

provided in Supplemental Digital Content 1.This self-administered questionnaire was de-signed to obtain information regarding thestructure and characteristics of the PICUs be-ing surveyed and the perceived value of bench-marking, public reporting, and pay-for-performance. In addition, we obtained moredetailed qualitative information regarding ex-periences and opinions with the existing CCS-mandated data sharing efforts. Three open-ended questions asked medical directors todiscuss their opinions and concerns about thefollowing: 1) the data being obtained, 2) meth-ods for data collection, and 3) the analysis andreporting of data. These topics were identifiedbased on initial formative discussion withstate officials who have worked on data shar-ing for several years, as well as several PICUmedical directors.

An invitation to participate in this surveywas initially mailed and e-mailed to all 19medical directors of CCS-approved PICUs dur-ing the summer of 2007. A research associatecontacted each medical director by telephoneto ensure the survey had arrived, extended aninvitation to participate in this informationgathering process, and explained to them thattheir responses would be confidential withoutany information identifying them or their in-stitution. If the medical director agreed toparticipate, then the research associate re-sponded to any questions regarding the self-administered instrument, which was returneddirectly to the study research associates bymail or fax. An appointment was also sched-uled for the semi-structured interview, gener-ally within the next week. Two of the studyteam researchers and authors (R.H., L.B.),made contact with medical directors, collectedall study instruments, performed phone inter-views, removed institutional/respondent iden-tifiers, and assigned a study code to all mate-rials to ensure anonymity for respondents. Asingle team member (R.H.) transcribed all in-terviews verbatim. These researchers nevermet or previously had spoken with any of thePICU medical directors.

Data Analysis. All data, both quantitativeand qualitative, were stripped of identifiersand analyses were conducted blindly. Descrip-tive statistics and univariable statistics wereconducted to provide means and medianscores on the Likert scales. SAS 9.21 andSTATA 9.1 were used for statistical analysis.Four investigators (L.B., J.J., J.M., and J.N.)reviewed all transcripts independently andidentified provisional themes. Final themeswere defined through an iterative process ofdiscussion, with subsequent annotation ofeach transcript with respect to these themes.The protocol was approved by the University ofCalifornia Davis Institutional Review Board.

RESULTS

Through postal service and fax trans-missions, 16 of the 19 CCS-approvedPICU medical directors completed the

Quality Improvement and Benchmarkingsurvey (84% participation rate). Threemedical directors who did not participatereported that they were too busy to com-plete the survey.

Participant Characteristics. Partici-pant PICU directors were from commu-nity, academic, and university-affiliatedhospitals, whereas the three nonrespon-dent PICU directors were exclusivelyfrom university-affiliated hospitals. Oneparticipating PICU (6%) was not affiliatedwith a medical school and was not ateaching hospital. Residents participatein educational rotations in 14 (87%) ofPICUs, and six (37%) have pediatric crit-ical care fellowship programs. Seven(44%) respondents characterized theirhospital as “public.” Eight (50%) of thePICUs had an in-house attending at alltimes. The PICUs varied in patient vol-ume, unit size, pediatric intensivist staff-ing, and average length of stay, as dis-played in Table 1. Sixty-nine percent ofthese PICUs described Medi-Cal as theirmajor payor.

Quantitative Data: Attitudes TowardBenchmarking, Public Reporting, andPay-for-Performance. Figure 1 displaysthe medical directors’ responses to ques-tions regarding benchmarking. Overall,73% of directors agreed or stronglyagreed that benchmarking improves pa-tient outcomes, and most disagreed orstrongly disagreed that it leads to avoid-ance of higher-risk patients (75%), higher-risk procedures (75%), or that it under-mines physician autonomy (67%). Medicaldirectors appeared ambivalent regardingthe validity of benchmarking. Notably,there was marked lack of unanimity re-garding the need for California-widePICU benchmarking and whether bench-marking provides accurate informationabout patient satisfaction and patientoutcomes. Opinions were nearly evenlydivided regarding other potentially ad-verse consequences of benchmarking,such as “a focus on statistics rather thanpatients,” “has negative repercussions,”and “leads to defensive medical practice.”

Figure 2 displays the medical direc-tors’ responses to questions regardingpublic reporting. Although medical direc-tors were divided about the potential ad-vantages and disadvantages of public re-porting, there was almost uniformagreement that report cards do not give acomplete picture of PICU quality of careand that publicly reported quality dataare misinterpreted. In accordance withtheir attitudes toward benchmarking,

2 Pediatr Crit Care Med 2011 Vol. 12, No. 5

AQ: 3

AQ: 4

AQ: 5

T1,AQ:6

F1

F2


60% of the PICU medical directors dis-agreed that public reporting leads toavoidance of higher-risk patients and pro-cedures. The responses to the statements“public reporting is punitive” and “publicreporting improves patient care in thePICU” illustrate the lack of consensusamong the medical directors. There weresimilarly diverse reactions to a statementregarding the need for public reporting ofPICU data in California.

Results regarding the perceptions ofpay-for-performance are shown in Figure

3. Again, these results demonstrated acontinued diversity of opinions amongthe PICU medical directors. For example,there were large differences in opinionswhen the medical directors were askedwhether physicians should be financiallyrewarded for better patient outcomes:seven strongly disagreed or disagreed,with an equal number strongly agreeingor agreeing (two were undecided).

Qualitative Data: Themes in MedicalDirector Interviews. Five themesemerged from the interviews with medi-cal directors: 1) value of comparative datafor internal benchmarking, 2) concernwith the perceived burden of data collec-tion, 3) persistent reservations about in-accurate data collection and reporting, 4)concerns that benchmarking data do notcontribute to quality improvement, and5) suggested approaches to data analysisand outcome measures.

Value of Data Collection for Bench-marking. Medical directors generally ad-vocated sharing quality performancemeasures across PICUs in the state. Re-sponses included, “It would be nice toknow more about how you are doing rel-ative to others” and “I do not have anytrouble with anyone seeing this kind ofdata.”

However, others reported concernwith public access to PICU performancedata while still valuing the use of com-parative data by the medical directorsthemselves. “… Until there is somestudy that shows how the public re-sponds to these data, I think it shouldstay within the institutions. We under-stand the variability and differences ofwhat we have within our institutions,so we can interpret variances a lot bet-ter than the public.”

Perceived Burden of Collecting Datafor Benchmarking. Directors were con-cerned that the currently mandated datarequirements are excessive, too time-consuming to collect, and require highlytrained and expensive personnel for accu-rate data extraction. Therefore, despitethe recognized value, PICU medical di-rectors expressed serious concern aboutthe burdens of data collection for state-mandated performance reporting. Be-cause of this, many recommended auto-population of the reporting data entrysystem with clinical information ob-tained from electronic medical records.Respondents also recognized that inac-curate data collection, whether by man-ual extraction or from electronic med-ical records, would lead to reported

outcomes that do not reflect the trueperformance of the PICU. Responses areas follows.

“Having an agreed upon set of vari-ables across ICUs would be important forcomparison. It cannot be 100 things, justa few which have real value, 6 or 7. Theyhave to be things that really can be use-ful to QI [quality improvement].”

“The biggest thing that stands in theway [of collecting desired information]is the number of hours it takes to col-lect it.”

“… The ideal situation is to have anICU physician [collect the required data],but it is very time-consuming … You justneed a certain amount of fundamentalknowledge about what goes on in the ICUto be reliable.”

An associated deterrent to data collec-tion for quality improvement was the fi-nancial burden, both for personnel andfor the software used to calculate theoutcomes, as shown by the following.

“I do know that the hospital spendstoo much on data collection. The govern-ing bodies tell us to collect data and donot fund anything … They should notask for more work without paying for it.”

Reservations About Bias. Medical di-rectors expressed frequent concern re-garding accuracy in data reporting. Inresponse, they provided several sugges-tions for increasing the reliability of thedata: auditing, third-party data collec-tion, standardizing variable definitions,quality-control checks, increasing auto-mation, and reducing human influenceon data collection.

“One thing that I am extremely con-cerned about … going to see catheter-associated infection go way down, notbecause of diminished prevalence, butbecause we will be reporting … in waysthat make us look better.”

“If pay-for-performance takes off,there is every incentive for people to mis-behave. We should audit the data entryprocess and there should be penalties forunits that have data entry fraud.”

Medical directors agreed that accuracyand reliability is essential to data quality.Recommendations for achieving thesestandards included standardized training,credentialing for data collectors, errorchecking and correction, and limitingdata entry to one or few highly trainedpersonnel with firsthand knowledge ofPICU care.

“A problem encountered is makingsure the quality of the data are the samefor all prospective sites, otherwise it is

Table 1. Pediatric intensive care unit character-istics as reported by medical directors

Characteristic

InstitutionsReporting

n %

Number of annualadmissions (2006)

�500 5 31%501–1000 4 25%1001–1500 4 25%�1500 2 12%Unknown 1 6%

Number of PICU beds(2007)

�11 7 44%11–15 2 12%16–20 3 19%21–25 3 19%�25 1 6%

Number of full-timepediatric intensivists

0 0 0%1 or 2 4 25%3 or 4 3 19%5–11 9 56%

Number of part-timepediatric intensivists

0 7 44%1 or 2 6 37%3 or 4 3 19%

Number of physicians andclinical full-timeemployees required tostaff PICU

2 or 3 44 or 5 56 or 7 4�7 3

Average length of stay(2006)

�3 days 2 12%3–4 days 5 31%�4 days 7 44%Unknown 2 12%

Payor mix for PICUpatients

Majority privateinsurance

2 12%

Majority Medi-Cal 11 69%Other payor mix 2 12%Unknown 1 6%

PICU, pediatric intensive care unit.


F3

AQ: 7


not helpful for comparisons and bench-marking.”

“I think it is important that you havea small number of people enteringdata … this is important for consistencyof reporting.”

Yet, given the resource limitationsfaced by these medical directors, suchgoals were often viewed as unattainable.

“… but basically because [of] the costof data collection … we have chosen tonot enter all the information we wouldlike, but cannot because of costs.”

Uncertainty That Performance Re-porting Contributes to Quality Improve-ment. As required by their CCS accredi-tation, all PICUs in this study report risk-adjusted mortality ratios to the State ofCalifornia annually. Medical directors ex-pressed widespread dissatisfaction withthis risk-adjusted mortality as a PICUquality measure for multiple reasons: 1)institution-specific characteristics (“It’sjust that we do not perform good in thisdatabase,” “Our population is fairlyunique … we end up looking terrible,”and “We have a large group on publicsupport and every study shows that peo-

ple like this often have worse outcomes.So how do you compare this group toothers?”) and 2) care received outside thePICU influences patient outcome yet isnot considered in risk-adjusted mortalitycalculation (“We are only measuring onepart of the hospital experience”).

Suggested Approaches to Data Analy-sis and Outcome Measures. Numerousand diverse suggestions were made by thePICU medical directors regarding alter-native methods of data analysis andmetrics of PICU quality. Furthermore,as one respondent observed, “I think itwould be valuable to have an opportu-nity for directors to have an input,rather than just a lot of bureaucracy.”Similarly another respondent reported,“If there are only 30 PICU medical di-rectors, why cannot we get togetherand look over these data together.” Be-cause their suggestions were so diverse,they are difficult to characterize collec-tively. Implicit in all of them, however,is skepticism regarding the adequacy ofcurrent approaches.

“I think our ability to get a handle onwhy a length of stay is protracted or

what are the various morbidities thatcontribute to a longer stay is not thereunless you go and extract it patient bypatient. I do not think we have a handleon that.”

Several alternative metrics wereviewed as potentially more useful qualitymeasures by the PICU medical directors.Suggestions for such quality measuresincluded the following: 1) hospital-acquired infection rates using commondefinitions; 2) condition-specific mortal-ity, length of stay, and cost data for com-mon diagnoses rather than all-cause; 3)unscheduled readmission rate; and 4) re-source utilization.

“We certainly look at bloodstream in-fections and ventilator-associated pneu-monia. I would like to see how our datacompares to other units and really honedown on the definitions because that isdifficult.”

“It would be nice to look at how thepatients are distributed in terms of ICUresources. For example, how many daysof mechanical ventilation, central lines,etc., to look more carefully at the utili-zation of resources.”

Figure 1. Attitudes of medical directors of pediatric intensive care units (PICUs) (n � 16) toward benchmarking based on a 5-point Likert score. Responsesrange from strongly disagree (left) to strongly agree (right). CA, California.



“I would like live data, so we couldreally be in a position to analyze ourcapacity throughout a region and usethat info to give patients access to care.”

One medical director advocated forpublic reporting of the benchmarkingdata.

“The public does deserve to know howan institution is doing before goingthere. If you are going to a place with astandardized mortality that is twice thenorm, you want to avoid that place. Thatis a little scary of a precedent, but hos-pitals need to fess up to this need.”

DISCUSSION

We report the attitudes and percep-tions of PICU medical directors regardingthree efforts to measure, incentivize, andimprove clinical performance: bench-marking, public reporting, and pay-for-performance. Although most CaliforniaPICU medical directors concur thatbenchmarking efforts improve patientoutcomes, they are uncertain that publicreporting and pay-for-performance canimprove quality of health care. Simulta-

neously, they are wary of the accuracyand validity of data used to generate theseperformance measures and are discour-aged by the time and costs required tocollect self information. These concernsextend to current requirements to pro-vide severity-adjusted PICU mortality.Leadership opinions appear potentially“dynamic” in multiple domains andacross each of the three measures as-sessed. Taken together, these findingsraise fundamental questions about howto create clinical performance standardsthat facilitate quality improvement in theface of a seriously divided constituencyand suggest further empirical research isurgently needed to document the effectsof such initiatives.

Quality improvement efforts havebeen credited with the identification ofbest practices (18), improvement of pro-cesses and outcomes of care across di-verse institutions (19), and better com-munication with consumers. However,quality improvement efforts and qualitycomparisons between doctors and hospi-tals have elicited concerns about physi-

cian autonomy (20), the quality of physi-cian-patient relationships (16), andaccuracy in conveying quality assess-ments to the public.

Benchmarking has been widely ap-plied across ICUs in recent years as partof more comprehensive efforts to im-prove quality of critical care, reduce med-ical errors, and minimize ICU costs (21–24). Severity-adjusted mortality rates andlength of stay between different ICUs is astandard benchmark for both adult andpediatric populations (25, 26). A criticalcomponent permitting the comparison ofbenchmarks across institutions is the ap-plication of risk adjustment. Fundamen-tally, such risk adjustment is requiredbecause the mix of patients seen differssignificantly across institutions and suchvariability contributes to differences inoutcomes. Because well-validated statisti-cal methods adjust for characteristics ofpatients who influence both the processesand outcomes of care but are not underthe control of the provider (27), risk-adjusted benchmarks can be legitimatelycompared across ICUs. Risk adjustment

Figure 2. Attitudes of medical directors of pediatric intensive care units (PICUs) (n � 16) toward public reporting of unit performance data based on5-point Likert score. Responses range from strongly disagree (left) to strongly agree (right).



typically includes adjustment for age,health status, socioeconomic status, andother patient factors. This allows out-comes such as mortality and length ofstay to be compared between heteroge-neous groups in an objective manner.

Although risk adjustment improvescomparability, our results indicated thatPICU medical directors expressed dissat-isfaction with risk-adjusted mortality as aPICU quality measure. As others have re-ported, consideration of patient comor-bidities and complications, as well as ac-curate coding and detailed clinical data,are essential for accurate risk adjustment(28). At the same time, these PICU med-ical directors join others in arguing thatthe costs of gathering the data necessaryfor risk-adjusted benchmarking may begreater than the benefits of facilitatingcomparisons across hospitals (29). Datafrom PICU medical directors confirmskepticism among regarding the cost datacollection and its potential validity (pri-marily focused on concerns about thatthe potential use to “game” the system inother institutions) (30 –32). Therefore,overcoming these concerns will be essen-

tial before PICU medical directors em-brace benchmarking data to motivatechanges toward patient quality improve-ment.

Studies demonstrating of the benefitsof public reporting efforts also have notresolved such uncertainty (7). Althoughsome have described quality improve-ment in institutions that participated in apublic reporting benchmarking initiative(33), there are also limitations, includingphysician concerns (5). Included amongthese are worries that patient data andresults can be manipulated by adminis-trators and practitioners, making accu-rate comparison impossible (34). Onesuch manipulation is the avoidance ofhigh-risk patients or procedures by phy-sicians. For example, Narins et al (20)found that surveyed cardiologists werereluctant to surgically intervene in high-risk patients, even though many felt thatpatients could benefit from such proce-dures. Another concern is that there maybe attempts to control or skew the pre-sentation or participation in public re-porting. For example, McCormick et al(35) found that in situations in which

Health Maintenance Organizations reportbenchmarking data on a voluntary basis,they are less likely to do so consistently ifthey are in a disadvantageous group. Datareported here found similar reservationsand concerns, even in a much more ho-mogeneous group of pediatric ICU medi-cal directors within a single state.

Notably, skepticism may be justifiedgiven the negative results of a large, re-cently reported, randomized Canadiantrial of public report cards as a method tostimulate improved treatment of myocar-dial infarction and congestive heart fail-ure (36).

The current investigation has beenundertaken in a state that is consideringdevelopment of new reporting measurescomparing PICU performance, and inwhich the opinions of the PICU medicaldirector constituency are, therefore, es-pecially pertinent. Data reported herestrengthen our understanding of PICUmedical directors’ attitudes by combiningquantitative and qualitative data. Thisunique effort successfully recruited PICUphysician leadership, experienced withstate-mandated reporting of severity-

Figure 3. Attitudes of medical directors of pediatric intensive care units (PICUs) (n � 16) toward pay-for-performance based on 5-point Likert score.Responses range from strongly disagree (left) to strongly agree (right).



adjusted mortality across a single largestate, thereby removing heterogeneityarising from variable child health andpolicy contexts. This makes it possible tobetter-appreciate how, under similar cir-cumstances, those in leadership respondto key issues relevant to quality improve-ment and cost containment. Such a re-stricted focus, which strengthens aspectsof our survey results, also imposes re-strictions in the variety, geographic loca-tion, and number of participants. Specif-ically, this survey was confined to PICUmedical directors in California and didnot assess the opinions and attitudes heldby staff intensivists, nurse directors, ormedical directors outside the state. Thus,our findings should not be used to repre-sent the broader attitudes and opinions ofPICU leadership nationally.

CONCLUSIONS

Our data demonstrate the urgency offurther research to document the haz-ards and benefits of quality improve-ment methods such as benchmarking,public reporting, and pay-for-perfor-mance. Accurate information about thestrengths and limitations of such ap-proaches can do much to address thecontinuing uncertainty and concerns ofphysician leaders. Our data also high-light the importance of including PICUleadership in designing data collectionand reporting systems for the purposesof quality improvement. Their sugges-tions would help inform recommenda-tions regarding data analysis and re-porting, the choice of metrics, and howreports should be structured. In addi-tion to providing substantive input,such participatory collaboration may domuch to overcome lingering suspicionsand concerns about initiatives such aspublic reporting of benchmarking data.Finally, this report highlights the needfor educating the next generation ofphysicians and physician leaders in themethods of quality improvement. Thepractice of medicine is changing rapidlyand the responses from the PICU med-ical directors demonstrate their ambiv-alence and uncertainty in these qualityimprovement efforts.

ACKNOWLEDGMENTS

We are grateful to the California PICUmedical directors who provided theirtime for these interviews and to all thosewho provide care to critically ill children.

Our thanks go to the State of California’sDepartment of Health Care Services, whoprovided guidance and funding for thisstudy.

REFERENCES

1. Halpern NA, Pastores SM: Critical care med-icine in the United States 2000–2005: Ananalysis of bed numbers, occupancy rates,payer mix, and costs. Crit Care Med 2010;38:65–71

2. Glance LG, Osler TM, Dick A: Rating thequality of intensive care units: Is it a functionof the intensive care unit scoring system?Crit Care Med 2002; 30:1976–1982

3. Egol A, Shander A, Kirkland L, et al: Pay forperformance in critical care: An executivesummary of the position paper by the Societyof Critical Care Medicine. Crit Care Med2009; 37:2625–2631

4. Chassin MR, Hannan EL, DeBuono BA: Ben-efits and hazards of reporting medical out-comes publicly. N Engl J Med 1996; 334:394–398

5. Epstein AM: Public release of performancedata: A progress report from the front. JAMA2000; 283:1884–1886

6. Lee TH, Meyer GS, Brennan TA: A middleground on public accountability. N EnglJ Med 2004; 350:2409–2412

7. Marshall MN, Shekelle PG, Leatherman S, etal: The public release of performance data:What do we expect to gain? A review of theevidence. JAMA 2000; 283:1866–1874

8. Epstein AM: Rolling down the runway: Thechallenges ahead for quality report cards.JAMA 1998; 279:1691–1696

9. Hannan EL, Kilburn H Jr, Racz M, et al:Improving the outcomes of coronary arterybypass surgery in New York State. JAMA1994; 271:761–766

10. Mukamel DB, Mushlin AI: Quality of careinformation makes a difference: An analysisof market share and price changes after pub-lication of the New York State Cardiac Sur-gery Mortality Reports. Med Care 1998; 36:945–954

11. Mukamel DB, Mushlin AI: The impact ofquality report cards on choice of physicians,hospitals, and HMOs: A midcourse evalua-tion. Jt Comm J Qual Improv 2001; 27:20–27

12. Dafny L, Dranove D: Do report cards tellconsumers anything they don’t know al-ready? The case of Medicare HMOs. Rand JEcon 2008; 39:790–821

13. Dranove D, Sfekas A: Start spreading thenews: A structural estimate of the effects ofNew York hospital report cards. J HealthEcon 2008; 27:1201–1207

14. Greene SE, Nash DB: Pay for performance:An overview of the literature. Am J Med Qual2009; 24:140–163

15. Luft HS, Garnick DW, Mark DH, et al: Doesquality influence choice of hospital? JAMA1990; 263:2899–2906

16. Werner RM, Asch DA: The unintended con-

sequences of publicly reporting quality infor-mation. JAMA 2005; 293:1239–1244

17. Fung CH, Lim YW, Mattke S, et al: System-atic review: The evidence that publishing pa-tient care performance data improves qualityof care. Ann Intern Med 2008; 148:111–123

18. Knaus WA: Improving intensive care perfor-mance: Complex paths to simple truths.Mayo Clin Proc 2005; 80:164–165

19. Afessa B, Kennedy CC, Klarich KW, et al:Introduction of a 14-hour work shift modelfor housestaff in the medical ICU. Chest2005; 128:3910–3915

20. Narins CR, Dozier AM, Ling FS, et al: Theinfluence of public reporting of outcome dataon medical decision making by physicians.Arch Intern Med 2005; 165:83–87

21. Becher EC, Chassin MR: Improving quality,minimizing error: Making it happen. HealthAff (Millwood) 2001; 20:68–81

22. Halpern NA, Pastores SM, Greenstein RJ:Critical care medicine in the United States1985–2000: An analysis of bed numbers,use, and costs. Crit Care Med 2004; 32:1254 –1259

23. Sirio CA, Shepardson LB, Rotondi AJ, et al:Community-wide assessment of intensivecare outcomes using a physiologically basedprognostic measure: Implications for criticalcare delivery from Cleveland Health QualityChoice. Chest 1999; 115:793–801

24. Zimmerman JE, Kramer AA, McNair DS, etal: Intensive care unit length of stay: Bench-marking based on Acute Physiology andChronic Health Evaluation (APACHE) IV.Crit Care Med 2006; 34:2517–2529

25. Pollack MM, Patel KM, Ruttimann UE:PRISM III: An updated Pediatric Risk ofMortality score. Crit Care Med 1996; 24:743–752

26. Zimmerman JE, Kramer AA, McNair DS, etal: Acute Physiology and Chronic HealthEvaluation (APACHE) IV: Hospital mortalityassessment for today’s critically ill patients.Crit Care Med 2006; 34:1297–1310

27. Kuhlthau K, Ferris TG, Iezzoni LI: Risk ad-justment for pediatric quality indicators. Pe-diatrics 2004; 113:210–216

28. Iezzoni LI, Foley SM, Daley J, et al: Comor-bidities, complications, and coding bias.Does the number of diagnosis codes matterin predicting in-hospital mortality? JAMA1992; 267:2197–203

29. Krumholz HM, Brindis RG, Brush JE, et al:Standards for statistical models used for pub-lic reporting of health outcomes: An Ameri-can Heart Association Scientific Statementfrom the Quality of Care and Outcomes Re-search Interdisciplinary Writing Group: Co-sponsored by the Council on Epidemiologyand Prevention and the Stroke Council. En-dorsed by the American College of Cardiol-ogy Foundation. Circulation 2006; 113:456–462

30. Bradley EH, Holmboe ES, Mattera JA, et al:Data feedback efforts in quality improve-ment: Lessons learned from US hospitals.Qual Saf Health Care 2004; 13:26–31



31. Campbell SM, McDonald R, Lester H: Theexperience of pay for performance in Englishfamily practice: A qualitative study. Ann FamMed 2008; 6:228–234

32. Casalino LP, Alexander GC, Jin L, et al: Gen-eral internists’ views on pay-for-performanceand public reporting of quality scores: A na-tional survey. Health Aff (Millwood) 2007;26:492–499

33. Hibbard JH, Stockard J, Tusler M: Does pub-licizing hospital performance stimulate qual-ity improvement efforts? Health Aff (Mill-wood) 2003; 22:84–94

34. Schneider EC, Epstein AM: Influence of car-diac-surgery performance reports on referralpractices and access to care. A survey of car-diovascular specialists. N Engl J Med 1996;335:251–256

35. McCormick D, Himmelstein DU, Woolhan-dler S, et al: Relationship between low qual-ity-of-care scores and HMOs’ subsequentpublic disclosure of quality-of-care scores.JAMA 2002; 288:1484–1490

36. Tu JV, Donovan LR, Lee DS, et al: Effectivenessof public report cards for improving the qualityof cardiac care: The EFFECT study: A random-ized trial. JAMA 2009; 302:2330–2337



JOBNAME: AUTHOR QUERIES PAGE: 1 SESS: 3 OUTPUT: Mon Oct 25 05:40:10 2010/balt5/zk8�pcc/zk8�pcc/zk800511/zk83766�10z

1—Please provide key words if those supplied are not OK.

2—Please verify ‘that are a means‘ as edited here

3—note supplemental material here; I don’t have documentation necessary to add link or title pagefootnote.

4—Please provide manufacturer information (manufacturer name and location: city andstate/country) for all nongeneric products mentioned.

5—Would “in-hospital” be preferred to “in-house” (noun) here?

6—Please carefully review all tables and figures, including figure captions and added figurecolumn heads. Correct that % values were not intended for “Number of physicians andclinical full-time employees required to staff PICU”?

7—Please verify quotes as edited. OK that stand-alone quotes are in italics but those within aparagraph are roman? Please make any necessary changes throughout.

8—Please verify correspondence and affiliation information are correct.

9—Please verify that funding statement is correct and complete

10—Please confirm that the conflict of interest disclosure statement is accurate and complete asshown for all authors.

AUTHOR QUERIES

AUTHOR PLEASE ANSWER ALL QUERIES 1

Documents

Benchmarking, public reporting, and pay-for-performance: A mixed-methods survey of California pediatric intensive care unit medical directors