Surgical Report Cards

----.~----------------

o SURGICAL REPORT CARDSTHE MYTH AND THE REALITY

Alan HendersonCorrespondence to Alan Henderson : [email protected]

There seems no good reason fordoctors towork insecret. Individual users ofhealthcare and thecommunity ingeneral. which ultimately bears the cost, are perfectly entitled toknow how theirhealth services and health providers are performing. The promulgation of surgical report cardshas been hailed by some as a liberating step inthe right direction.This paper seeks toanalyse,from a clinician's perspective, the evolution and limitations of report cards. Ultimately, the importance of report cards will not be their immediate utility, which isminimal, but as afirststepinamuch wider and farmore important debate about how wemeaningfully measure the qualityofhealth services and providers (including managers and bureaucrats), the likely cost ofsuch anenterprise, how much weare willing and able topay and how wereconcile the competing needs 'of information versus clinical and preventive care when allare competing forthe same and inadequate pool of resources.

INTRODUCTIONThe publishing of standardised cardiac surgical mortality data in the public domain (also

known as 'report cards'), particularly in the USA and the UK, has been met with interestand enthusiasm amongst the bioethics community, hyperbole by the newspapers - wherein the UK extravagant claims have been made that report cards have caused a drop incardiac surgical mortality - and with a mix of indifference and suspicion from clinicians(Boseley 2009). The difference, I suspect, relates more to a different world view rather

than to any deeply held difference in values. Much has already been written on thissubject by bioethicists, so I intend to examine the issue of report cards with the hermeneutic lens of a clinician (cardiologist and intensivist), briefly tracing their developmentand then looking at some of the major issues which severely limit their value and whichmay pass unnoticed by non-clinicians. Finally, it is often forgotten that information has,

like any commodity, a cost . I will use an example from my intensive care practice toshow just how expensive good quality data can be, and I will use this to reflect uponimplications for the Australian health system if the Australian government were tomandate the collection by all doctors of meaningful performance outcome data.

20.1 MONASH BIOETHICS REVIEW. VOLUME 28.NUMBER 3.2009 MONASH UNIVERSITY EPRESS

THE RISE OF MEDICAL TRANSPARENCY

Report cards are part of a wider trend towards medical transparency which seeks to

measure and promulgate the outcomes of medical interventions, thereby demonstrating,rather than assuming, that doctors are performing well, are not using scarce resourcesprofligately and in general are worthy of the trust placed in them. The perceived needfor medical transparency comes, in part, from a more general decline in trust in all aspectsof authority seen in Western style democracies over recent years and the rise of its oppos

ite, suspicion, fostered by the twin bedfellows of spin and deception . This has been referredto as a "crisis of trust" (O'Neill 2002: 4-10). The very idea of a crisis of trust taking

hold in our society should be viewed with concern, given that trust is the glue whichholds our communities together. The pivotal role of trust was recognised centuries agoby Confucius, who said of the three great pillars of government; trust, weapons and

food, trust was the most important and should be the last to be forsaken (Waley 1938:164).

One way governments, businesses, institutions and to some extent even individuals

have responded to the decline in trust and the increase in suspicion, is to place informationabout their activities (more accurately, some of their activities) in the public domain.Until recently, this was a difficult and laborious business, but with the ready availability

of powerful, cheap computers and the development of the internet, vast amounts of information can be easily collected and disseminated, as evidenced by the copious companyand government reports which are now available. Indeed, the convergence of advances

in information technology with a decline in trust have together conspired to give birth

to the transparency revolution.Medicine has not remained aloof, although it is a late comer. I suspect that the spread

of transparency into clinical practice was not so much the result of deep philosophicalor ethical deliberation, but rather a response to this pervasive decline in trust in the widercommunity, fuelled, on occasion, by media generated scandals. The response to scandalis often spin which, if cleverly disguised, easily passes for a genu ine concern for patient

safety.The drivers for medical transparency, at least with regard to adult heart surgery,

where experience is greatest, are different in the USAcompared with the UK and Australia .In the USA, the initial clue that a problem existed came from cardiac surgeons review

ing their own data. A survey of mortality after coronary artery bypass grafts (CABG)between 1991 and 1993 involving over 600 hospitals and over 1200 surgeons revealed

excess standardised mortality in hospitals undertaking under 100 CABG operations an-

SURGICAL REPORT CARDS ARTICLES 20.2

nually (Clarke 1996). Subsequently, various health authorities led by New York Statehave published standardised surgical mortality rates for hospitals and individual card iacsurgeons (New York State Department of Health 2008). CABG surgery has become thefocus of the medical transparency movement partially for economic reasons (1000/millionoperations annually in the USA and 700/million in Australia) and partially for pragmatic

reasons in that it is the most intensely studied of any operation of surgery and it has ahas a fairly constant mortality of 1%-2%(Warrel et al. 2003 : 952) . Mortality, I suspect,was chosen as a qua lity indicator because of its obvious importance and because it isunambiguous and easy to study. It is important to remember, however, that othermeasures of quality exist, which are as yet below the public radar.

In the UK and Australia, the drive to transparency has been scandal. Governmentsof all complexions hat e and fear public scandal, particularly in voter sensitive areas suchas health . The response to a serious scandal ar ising in a domain of government responsibility often is stereotyped, a phenomenon recognised by MacDonagh in the nineteenth

century (MacDonagh 1958). Typically something bad is happening, the public (oftenvia the media) become aware and an embarrassing public scandal follows. Governmentthen attempts to allay public concern and regain trust by enacting legislation.

The goings-on in England, at the Bristol Royal Infirmary paediatric cardiac surgical

unit , illustrates th is in action. Here, over a long per iod, two surgeons had poor resultsfor certain operations. This was well known to some of the clinicians in the hospital andto hospital management, but hidden from public view. All was eventually revealed by awhistle blower. There was a media frenzy, a terrible public scandal and gross anger and

embarrassment within government circles. Something needed to be done and be seen tobe done to regain public confidence in the NHS . A Commission of Inquiry chaired bySir Ian Kennedy was charged to put things right, and its recommendations for greatermedical transparency and accountability accelerated moves to publish surgical outcomesdata (Sorell 200 7). The British Care Quality Commission now publishes mortality data

for all adult cardiac surgical units and for individual cardiac surgeons (Care QualityCommission 2008).

Australia had its own scandal with general surgeon jayant Patel in Bundaberg,

Queensland . The pattern was very similar to Bristol and the result identical. A Commissionof Inquiry (the Forster Inquiry) reported in 2005 and recommended increased publicaccountability for surgeons.

At least in the UK and Australia, the motives for increased medical transparencyseem to be the need for government to show the public that something is being done tocorrect a wrong and to regain public confid ence in the healthcare system. The very notion

20.3 SURGICAL REPORT CARDS ARTICLES

that more transparency is the solution to the problem is, however, a bureaucratic way

of looking at the world . While increased public confidence based upon published datamight be the prime motivation for spin doctors, I suspect medical doctors and their patients might be more concerned with improved standards of care as a basis for increasedconfidence in their healthcare system. One important question to be answered is: do reportcards improve clinical outcomes? This question is more complex than it at first appears

and requires an understanding of a wide range of issues which impact on the publishedmortality rate.

SURGICAL MORTALITY DATA AND CLINICAL OUTCOME

If the reporting of mortality data could be shown to improve clinical outcomes, then astrong case could be made to support it and indeed to extend it to all branches of clinicalpractice. We need to be careful not to rush to overhasty judgments, given that reporting

is still in its infancy and its full effects have yet to be revealed.

AlCAUSE OR CORRElATION?

It is crucially important to not confuse correlation with a causal relationship, as appears

to be the case in the UK media. A correlation occurs when two variables appear to movein relationship to one another, but this movement is either a chance occurrence or the

influence of hidden third variable which is related to the other two . For example, it mightbe possible to show a parallel fall in deaths from heart disease and the fall in broadsheetnewspaper sales. Surely, no one would suggest a causal relationship existed. Before weascribe causality to the relationship between falling death rates and report cards, we

must have a plausible hypothesis connecting the two variables and evidence to support

cause rather than correlation.What would a causal hypothesis look like? If simply putting mortality data in the

public arena reduced deaths then one would need to put forward the view that cardiacsurgeons were operating, protected by anonymity, in a manner heedless to the fate oftheir patients. Aside from the very rare cases of rogue surgeons, is there any evidence forthis? Certainly my experience as a cardiologist working with many surgeons over thirtyyears leads me to believe that the vast majority of surgeons are deeply concerned withpatient safety and high quality care and are jealous of their professional reputations.Heedless practice would need to be widespread if report cards were to make a measurableimpact on mortality and there seems little to suggest that is the case. What of the evidence

of causality?

SURGICAL REPORT CAROS ARTICLES 20.4

B) EVIDENCE THAT REPORT CARDS IMPROVE CLINICAL OUTCOMES

The quality and quantity of dat a is rather poor, but overall, the literature is rather un

convincing as far as showing report cards favourably influence clinical outcomes. TheAmericans, who have the longest experience, have produced a number of studies. Soonafter the publ ication of cardiac surgical outcomes in New York State a drop in post-operative mortality of 41 % was reported (Marasco & Ibrahim 2007). This seems truly remarkable. In clinical practice, a randomised tr ial producing a benefit of 20%-25% would

be considered highly significant. A difference as large as 41% over such a short periodwould be most unusual, particularly as the only change seemingly was the publishing ofreport cards, rather than more obvious determinants of outcome, such as improved surgical training, improved access to advanc ed technology and post-operative care. Results

which look too good to be true need to be approached with caution and a search madefor confounding variables. In th is case might the drop in mortality have resulted fromsurgeons turning away high risk pat ients? The question was not asked so the answer isunknown.

Before and after studies from elsewhere, however, do suggest that report cards mayhave changed surg ical behaviour and that at least some of the drop in death rates relatedto surgeons avoiding higher risk patients. Dranove et al. used Medicare data for NewYork State and Pennsylvania and found a fall in acuity levels (surgeons avoiding high

risk patients), increased costs and worse outcomes for sicker patients after the introductionof report cards (Dranove et al. 2003). Werner et al., reporting from New York State,

found that there was a fall in cardiac operations on African-Americans and Hispanicsafter the introduction of individual report cards (Werner et al. 2005). Patients from theseethnic groups are generally considered higher-risk patients because of generally poorercompliance with therapy and overall poorer health status compared with whites (Werneret al. 2005) . These findings raise grave concerns about the effect of report cards on dis

tributive justice. In the USA certain ethnic groups, including African-Americans andHispanics, are burdened by poorer cardiovascular health, poverty and social deprivation(Bonow 2005) . It seems report cards may be further ing this injustice. Aside from cardiacsurgery a study of Nursing Home report cards showed no benefit in terms of raisingstandards of care, while a recent review of all studies of the effects of public reportingof outcome data since 1986 found no evidence of any benefit in terms of quality of care,patient-ccnteredness of care, or of patient safety (Werner & Asch 2007; Fung et al. 2008) .The evidence that report cards cause a fall in mortality seems rather feeble, so what then

20.5 SURGICAL REPORT CAROS ARTICLES

are we to make of the hyperbolic report in the Guardian (UK) newspaper which claimsthe opposite (Boseley 2009)?

Firstly, it is important to remember that the healthcare systems differ and that theBritish National Health Service (NHS) is quite different to the US Healthcare system, soextrapolating US dam to the NHS must be done with care . In the UK, the publicationof adult cardiac surgical mortality data followed the Bristol scandal and may be a classicexample of the MacDonagh phenomenon. Since the publication of institutional (not individual surgeon) mortality data there has been a 20-30% fall in death rates . This is notdisputed, but the question arises whether we are seeing a causal relationship and if so,how powerful a causal relationship, or just a correlation. Data from the NHS provides

a clue. Heart disease has long been recognised as a priority area because of the highburden of disease and poor outcomes. In response, the British Government establisheda National Service Framework for Coronary Heart Disease (NSFCHD) which , in 2000,set a target of a 40% reduction cardiac deaths to be achieved by 2010 (Department of

Health 2004). A massive effort with intensive funding across a number of areas of cardiachealth was forthcoming. This included preventive health in areas of exercise, smoking,

obesity, diabetes, hypertension and diet, improved access to out-of-hospital and EmergencyDepartment thrombolysis for heart attacks, employing more cardiologists, nurses and

technicians to facilitate improved access to angioplasty, including infarct angioplasty,more cardiac surgeons and extension and expansion of card iac surgical centres and soforth. Between 1995 and 2002, a fall in cardiac deaths of 23% was recorded (Departmentof Health 2004). Cardiac death rates were falling quite sharply well before Bristol. Anupdated report from 2008 shows a fall of 44% in cardiac deaths in the UK with a remarkably constant year-by-year decline (Department of Health 2008). If report cards

had an influence it is hard to sec it from the data. Overall, the very substantial improvement in cardiac care in the UK is almost certainly due to the enormous programme ofimprovements wh ich commenced before report cards and which has been dramaticallysuccessful in exceeding its own target of a 40% reduction in deaths by 2010. The effectof individual interventions was not measured so it is not possible to say with certaintywhat, if any, impact institutional report cards had . Most likely, given the enormous increase in resources poured into cardiac health, it was at least relative to them, probablynegligible . Of course report cards might have utility beyond reducing death rates , forexample by providing information for improved decision-making, but do they achieve

this?


REPORT CARDS AND INFORMED DECISION-MAKING.Few, I think, would argue that provid ing doctors and patients with accurate informationto assist in making healthcare choices is anything but laudable. Providing poor or incomplete information, however, merely leads to confusion and detracts from informed decision-making. This leads to the question as to what precisely does a published deathrate mean? I will focus on the British NHS .

Suppose a British card iac centre publishes a death rate for valvular heart disease as2.5% . Is this a genuine attempt to inform the public and to ensure high quality care, oris it a MacDonagh effect?

Having a clear understanding of risks and benefits is fundamentally important whenit comes to patients confronting the possibility of heart surgery. A number of potentiallyconfounding factors need to be addressed before we ascribe any value to standardiseddeath rates.

A)DO REPORT CARDS IMPROVE PATIENT CHOICE?

Except for the very wealthy, heart surgery in the UK is available only through a socialisedmodel of health care via the NHS. British cardiac surgical centres have traditionally been

plagued by long waiting lists. Reducing waiting times to (only) six months from twoyears has been hailed as one of the great achievements of the NSFCHD (Department of

Health 2004) . Further, cardiac services in UK are regionalised, so patients from anyparticular NHS Region will go to the cardiac centre serving that Region. It is impossibleto believe that a patient from one Region would be accepted by another Region, itselfstruggling to meet the needs of its own patients, just because the patient reviewed themortality data and decided not to risk an operation in his or her own Region. In reality,NHS patients will be referred to their own Regional centre and have no choice in the

matter (and no choice of surgeon either) .Where a substantial private sector exists, such as in the USA or Australia, well-in

formed, wealthy and socially empowered patients might take advantage of report cardsto choose better surgeons, leaving the poor, disempowered and often less healthy patientswith the poorer surgeons. I doubt that this affront to distributive justice was intended,but it seems a natural consequence of the play of market forces. All th is assumes potentialpatients use the data available.


BI DO PATIENTS USE REPORT CARDS WHEN MAKING HEALTH CHOICES?

The data is weak. From a personal perspective, I found the NHS website rather difficult

to find and although it looks attractive and is meaningful to someone familiar with theinterpretation of medical statistical data, I suspect those who are less skilled, older, lesseducated and ill might find it rather opaque, although I am not aware that this has beenformally tested (Care Quality Commis sion 2008).

Studies from the USA found that when it comes to making health choices, patientsseek advice from the ir doctors, family and friends and hardly any (only 12% inPennsylvania) used or had even heard of published surgical outcome data (CanadianHealth Services Research Foundation 2006; Schneider & Epstein 1998). Maybe theproblem lies in the presentation, rather than the concept. Finding out what patients want,

rather than what bureaucrats imag ine they need, might be a good start. This assumesthat a mortality rate has utility when making heal th cho ices.

WHAT DOES AMORTALITY RATE MEAN?

What, precisely, does a mortality rate of 2.5% mean? We know that outcome dependson a host of variables, especially how old and sick patients are (burden of co-morbidities),

the complexity and severity of the index illness, the availability in a timely fashion ofappropriately sophisticated medical facilities and the skill of the healthcare team . Crudesurgical mortality data (percentage of deaths related to all those treated), because of

these confounding variables, tells us little or nothing about the skill of the surgeon. Forthis reason mortality data are standardised, that is, account is taken of a range ofphysiological variables, such that the impact of .health providers on outcome can beisolated and measured. It is easy to assume that once data is standardised accurate interdoctor or inter-hospital comparison can be made. Unfortunately, the reality is somewhat

different.In the area of cardiac surgery alone, multiple scoring tools (at least 19) have been

developed, suggesting none is perfect . An important and widely used standardisationscoring tool to estimate operative mortality risk is the EuroSCORE. Because the EuroSCORE is so widely used, it provides a good platform to discuss issues with standardisation . It has long been thought that the EuroSCORE overestimates risk and may lead topatients being assigned higher risk categories than their illness merits or denied surgeryaltogether. Recently a study from Canada has illustrated the magnitude of the problem.Kalavrouziors et al. evaluated EuroSCORE predictions against actual early and latemortality for patients with severe aortic stenosis (Kalavrouziotis et al. 2009). They found

SURGfCALREPORT CARDS ARTICLES 211.8

the EuroSCORE overestimated the risk of death by a factor of 3. Does this matter if

everyone is using the same tool? Unfortunately it does, as there may well be variation ingatekeeper (the doctors deciding who is to be advised to have cardiac surgery) sensitivityto the EuroSCORE. Doctors with low sensitivity to the score (those who are less trustingof its predictions) will operate on higher risk patients than those more trusting of thepredictions, who may refuse surgery to pat ients with high scores . Further, it appears thateven patients with the same scores are not adequately standardised because of subtleinter-institutional differences in the definitions of the variables upon which the scoredepends (Nilsson et aI. 2006). So when we compare one surgeon or hospital with anotherwe need to be alert to the limitations of current standardisation tools which may make

direct comparisons less secure than they at first appear. Patients moving in and out ofhigh risk catego ries derived from the score can have a big impact upon standardisedmortality rates . Further problems lurk below the surface.

AITEAMS OR INDIVIDUALS?

If we find a larger than expected mortality rate, should we blame the surgeon? It could

be a statistical chance event, a problem with imperfect standardisation for severity ofillness, it might be related to a higher risk population, or to the care given. With regardto the standard of care it is important to note that modern medicine is a complex team

activity and that card iac care is particularly complex. Clearly surgical skill and judgementplay an important role in determining outcome, but so do the skills of the medical cardiologist, the theatre team (nurses, pump techn icians, anaesthetist), the Intensive Care

service (a very complex team in itself), ward care and rehabilitation. Poor outcomes canemerge from deficiencies anywhere in the system so if higher than expected death ratesoccur, it shou ld precipitate a search for the reason, rather, as may be assumed, that it isthe surgeon at fault.

B) ISA COMPOSITE MORTALITY RATE USEFUL?

A published mortality rate of 2.5 % for valvular heart disease tells us surprisingly little.Does it refer to aortic valve replacement (with or without aortic root surgery), mitralvalve replacement, mitral valve repair, tricuspid annuloplasty (occasionally replacement),pulmonary valve surgery, valve surgery combined CABG or any combination of theabove, each with their own set of risks? What the .pat ient really needs to known is themagnitude of his or her indiv idual risk for a var iety of outcomes important to that particular person (which aside from death might include , stroke risk, infection risk, the needfor Warfarin, pain , cosmetic effect of scars and so forth) . The composite rate might be


2.5% but because of technical issues, which cannot be standardised, the individual risk

for death might be 15% and for stroke 20%, Here a frank discussion with the cardiologist

and cardiac surgeon is required rather than perusal of an average death rate.

It might be claimed that average death rates would detect the rare rogue surgeon.

They probably would, but rogue surgeons represent not only clinical and moral incom

petence, but a gross failure of clinical governance and these people might be better detected

and eliminated by improvements in clinical governance rather than by report cards.

C) REPORT CARDS AND THE OVERALL QUALITY OF CARE

Published mortality data tell us little about the overall quality of the healthcare because

they provide such incomplete information. They give some information about what

happens to patients who undergo surgery, but if we want to know whether our healthcare

system is itself healthy, we need to ask questions about the fate of patients who might

benefit from surgery, but do not have an operation (refuse surgery, are refused surgery

or die on the waiting list), especially if there are large numbers of them. Alas the published

information is depressingly scanty.

I will use aortic stenosis (AS) as an example, as there is some recent data on this.

Severe AS is common and the prevalence increases sharply with age. In Western countries,

3% of the population over 70 years have severe AS (Chambers 2005) . Many patients

present at an advanced age with very severe AS and multiple co-morbidities which render

then either at high risk for Aortic Valve Replacement (AVR) or too ill for AVR. In my

cardiology practice about 20 % of those with severe AS would be unsuitable for AVR.

Turning down patients for surgery is a major decision , as there is no effective medical

treatment for the condition and we know that the prognosis for patients with sympto

matic AS who do not have AVR is poor, with virtually all suffering an adverse event

within 5 years and between 50% and 85% dying (Vahanian et al. 2006). If the number

of patients being refused surgery is large, or if large variations exist in how the decision

about suitability for surgery is made, the n using mortality data of those operated on as

a yardstick for the quality of care is nonsense. Unfortunately no information about these

patients appears on government websites.I need to divert for a moment to relate how patients are selected for surgery. Patients

must pass a number of gatekeepers before reaching the operating theatre. Th is complex

web is often not fully understood and many believe it is the card iac surgeon who decides

who is and who is not operated upon . The first gatekeeper is the patient. Patients with

AS may not recognise symptoms or attribute them to ageing and therefore never present

for diagnosis. This may explain the relatively high frequency of previously undiagnosed


severe AS found at autopsy (Chambers 2005). Second comes the General Practitioner(GP). How many elderly patients with AS are seen by the GP but not referred for cardiological evaluation is unknown, but from my own experience, I suspect that it may besignificant. The GP will then refer the patient to a specialist , who mayor may not be acardiologist. An interesting study from Leeds in the UK showed that, astonishingly, only

20% of patients seen by a geriatric physician with severe AS confirmed by echocardiography, were referred for consideration of AVR (Abdul-Hamid & Mulley 1999). A largecommunity study from Finland showed similar results with only a minority of elderly

patients with ASwho could benefit from surgery being referred for an operation (Lindrooset al. 1993).

Ultimately all patients see a cardiologist, who will assess the patient's suitability forAVR. Some patients will be turned down for surgery at this stage - estimates vary, but

it is thought to be as many as 20%-30 %. How cardiologists go about making these decisions is unclear, although we know from a Dutch study that there is considerable systematic variation in how patients were selected by cardiologists for operation (Boumaet al. 2001) . This variability is a problem when attempting comparisons between surgeons

or hospitals. Eventually a minority of patients see a surgeon. Only a small number ofpatients referred by the cardiologist would be rejected at th is stage for technical reasons

or because of the burden of co-morbidities (Waterworth et al. 2008). It can be seen fromthis that with elderly patients with AS (who are the majority), only a small number willbe referred for operation and contribute to publ ished outcome data, giving a very pooroverall picture of the quality of care offered to patients with AS as a whole. When they

do undergo AVR,even very old patients do well with operative mortality rates of between5%-10% in patients over eighty (including some over ninety) (Kojodjoj et aI2008). Patients with similar co-morbidities and age who refuse the offer of AVR have a greaterthan twelvefold risk of dying compared with those having AVR. It is this sort of detailedinformation that patients need to assist their decision-making, although as I mentionedearlier, care needs to be taken with mortality prediction scoring tools such as EuroSCOREwhere a high score might lead to a patient being excluded from surgery when, in fact,they would benefit from AVR.

It can be seen from this gatekeeping cascade that, although the surgeon ultimatelydecides who has an operation, most of the decision-making is in fact made by others,well before the patient is ever seen by a surgeon. Some might find it surprising that thesurgeon is such a minor player in the gatekeeping proc ess. This needs to be borne inmind when issues of defensive surgical behaviour are cons idered. If report cards changesurgical behaviour (there is some evidence for th is) the overall impact might be smaller


than expected, at least for AS, as the surgeon is only called upon to make a decision ona small minority of patients. Also we need to be careful about what defensive surgery is.

It is most certainly not refusing to operate on high risk patients. Surgeons may turn downpatients desirous of surgery because of technical difficulties (for example, a patient withangina who has such severe coronary disease that the surgeon can find no vessels to graftinto), because of extremes of age (age itself is rarely a determining factor, but few surgeonswould operate on patients approaching or over 100), or because the burden of co-morbidities precludes a successful outcome. Turning down these patients is the exercise ofsound surgical judgment. It is quite different from the morally corrupt business of defensive surgery, whereby a higher risk patient who could benefit from surgery and who wantsan operation is turned down by the surgeon to protect a report card.

If simplistic mortality data fail to provide much useful information, might they yetbe useful in instilling public confidence in doctors and the healthcare system?

REPORT CARDS AND TRUST

We are led to believe that there is a general decline in the levels of trust and an increasein suspicion across all areas of society, "the crisis of trust" (O'Neill 2002: 4-10). Despitethis, hospitals are filled to overflowing and doctors remain as busy as ever. This wouldsuggest the community still has a high level of trust in its doctors, nurses and hospitals,

even though it may be wary of Health Departments. A MORI Poll conducted in the UKshowed that doctors topped the league as the most trusted (doctors 98%, teachers 86%,professors 78%, judges 78%, priests 73% and holding up the bottom, po liticians 18%)(MORI Poll 2007). The Poll shows where the problem of trust lies. The public, at leastin the UK and probably in Australia, simply do not trust politicians (or the bureaucratswho serve them, including the health bureaucracy). This is clearly a serious problem

when governments legislate to enforce the publishing of mortality data from their ownhospitals. Will the public trust healthcare stati stics from government-run hospitals anddepartments, collected by government employees and published by government civilservants after the attention of the appropriate spin doctor?

We have seen grave moral lapses in the behaviour of the hospital bureaucrats whowere charged with ensuring patient safety at Bundaberg and Bristol. An even moreshocking example of how politically motivated bureaucrats work ing for the British NHSfelt able to further their ambitions and serve their pol itical masters while plac ing thepatients under their charge at great risk came from a hospital in Stafford (Smith 2009).


There, corrupt managers developed reckless strategies in an attempt to comply with

government targets for Emergency Department (ED) waiting times. Critically ill patientswere removed from the ED without being properly assessed or treated to comply withexternally imposed time limits . When the scandal eventually broke, an inquiry foundthat as many as four hundred patients had needlessly died. Those responsible subsequentlyresigned , but it is this sort of behaviour, coming after and apparently uninfluenced bythe Bristol scandal, that has so undermined confidence in government health departmentsand their bureaucrats.

Nearer to home, the Commission of Inquiry into the Patel scandal in Bundaberguncovered a litany of questionable dealings by health bureaucrats, a culture of secrecyand bullying in the Health Department, including intimidation of those raising concerns

about standards of care (Davies 2005). Given the behaviour of (some) politicians andsome of the bureaucrats who serve them (to the detriment of the many honest and capablehospital managers), their low rating in the "trust report cards" seems largely justified,making it unlikely the public will readily trust any information coming from governmentdepartments, knowing that the data may have been manipulated by spin doctors forpolitical purposes.

This being said, the community has a right to know what its doctors are up to, espe

cially when the taxpayer funds all or most of the large costs of healthcare. The publicationof meaningful data would not on ly address this issue, but would assist doctors to improvethe ir own perfo rmance and assist patients in making truly informed decisions abouttreatment options.

MEANINGFUL DATA, BUT AT WHAT COST?Information, like any commodity, comes at a cost , and in the health sector we need to

buy info rmation just as we buy artificial hips or pacemakers. Before demanding moreand better information, we need to pause to ask the question: at what cost? How muchinfo rmation do we want, how much can we afford and what are we willing to forego inorder to buy the information? These questions seem sometimes to be forgotten , especiallywhen patient autonomy and informed choice is at stake. Remember also the scope ofthe problem. If publishing meaningful cardiac surgical audit data is thought to be a goodthing, then surely the practice needs to be rolled out to cover all the medical and surgical

specialties, psychiatry and General Practice . In Australia, this would mean auditing thework of over 62,000 doctors on an ongo ing basis (Australian Inst itute of Health andWelfare 200 7).


The implications are quite staggering. Let me illustrate the point by sharing my ownexperience of quality of care audit while I was working as the Director of Intensive Care(and one time Chairman of the Division of Medicine) in a large university teaching hospital.

Measuring the quality of care in the Intensive Care Unit (lCU) has always been achallenge due to the diversity of patients with a very wide variety of illness and injuriesadmitted to ICU, the complexity of the team environment where anyone patient mayover time be cared for by a number of ICU consultants and numerous nurses, and theheavy influence on outcome of events happening before, during and after admission toICU related to doctors independent of the ICU team (e.g. the quality of surgery, treatmentin the ED and post-ICU ward care) .

One important way of assessing quality is to compare outcome data with other ICUs,but this is hampered by the wide differences in the types of patients (casemix) admittedto Australian ICUs (Anderson & Hart 1988). A number of databases and standardisationtools have been developed, but none is perfect. Further, it can be impossible to adjust

for some casemix variables - for example, ICUs admitting large numbers of severelyburnt patients (with very long and complex admissions, many complications and highdeath rates) are impossible to compare with ICUs admitting mostly elective cardiac sur

gical patients (short stay, low mortality and complication rates). Also ICUs work in

networks, whereby smaller units transfer more complex patients to larger units, with aprofound impact on outcome data . Suppose, for example, ICU A has a death rate of 5%and ICU B a death rate of 15%. Before booking your berth in ICU A and closing downunit B, some severity of illness adjustments must be made. ICU A is a small regional unitthat transfers all its difficult cases to ICU B, in a university teaching hospital. Standardising

the mortality data for severity of illness produces a different picture. The expectedstandardised mortality for unit A is 2.5 % (actual 5%), wh ile the expected standardised

mortality for unit B is 20% (actual 15%).When confronting the notion of audit, careful thought needs to be given as to what

data should be collected and what in reality can be collected with the endpoint of havinginformation to inform future clinical practice, to demonstrate high quality, safe and cost

effective practice, and in many areas of medicine, although perhaps not quite so muchin critical care, to provide the right sort of information to patients to allow them to make

more informed choices about their healthcare.In the ICU environment, standardised mortality data, while important, are insufficient

to judge performance. Other variables which are generally held to be important are thingssuch as deaths after discharge from ICU, length of time in ICU, time ventilated, the


number of patients developing kidney failure, nosocomial infections, especially with

highly resistant bacteria, unexpected re-admissions, the number of patients receivingpotentially toxic agents such as gentamicin without the monitoring of blood levels, DVTprophylaxis and so on.

Having decided upon the quality indicators to be collected, the next issue was how

to standardise for variables which are known to influence outcome, such as age, type ofillness, and a range of physiological variables measured on admission. Traditionally, theAPACHE II severity of illness scoring tool has been used for this purpose (Knaus et al.1985). It has the advantages of being simple (it is a paper-based system), quick and cheapbut is rather dated, only modestly predictive and then only for groups of patients, 'notindividuals.

To circumvent these limitations 1 opted to use the APACHE III tool (Knaus et al.

1991) . This is a complex, computer based tool collecting a wider range of variables ona daily basis (APACHE II collects data only on admission to lCU). The data from allAPACHE III centres around the globe are integrated at a centre in the USA.The individual

centres are then given daily mortality and length of stay predictions for every patient(trend lines show if the patient is improving or deteriorating with treatment) and wellstandardised comparisons of a range of performance variables against those predicted

for patients with similar levels of illness treated in similar ICUs all over the world(Henderson et al. 1997).

This produced superb and meaningful information, but at a cost. The initial capitaloutlay for the computing technology was over $250,000 in 1993 with ongoing costs ofabout $50,000 per annum. Because of the complexity of the data set it soon becameapparent that not only wou ld 1need a full-time data collector, but that-the data collector

needed to have a good understanding of intensive care, requiring the employment of alevel II lCU nurse. Senior, experienced lCU nurses are at once scarce and expensive.

While 1am not putting this forward as a model for lCU audit, it did reveal the oftenhidden costs of buying high quality information. Not only are scarce and highly skilledstaff tak en from the bedside to manage data, but there is a recurrent cost to fund salariesand technology. My example represents one tiny part of healthcare - what if we were

to extrapolate this to all aspects of healthcare, in every hospital department, every general practice and every private doctors rooms? Huge amounts of effort would need tobe expended in developing and then validating appropriate audit tools. Large numbersof people (clerks, but also scarce health professionals) would need to be employed tocollect, analyse and interpret the data . Doctors and nurses are in short supply, so if theydivert their time to data, they divert it away from direct patient care, possibly adversely


affecting outcomes. The possibility that collect ing audit data harms patient care by di

verting resources from the bedside needs to be carefully investigated before any largescale programme is introduced. If deta iled aud it and information gathering is shown tobe beneficial, how is it to be paid for? It seems that all governments and pr ivate healthinsurers are struggling to fund the current healthcare costs of their citizens and members,

particularly as the population ages. If this potentially huge extra cost is to be met withinexist ing budgets, then a decision will need to be made as to which areas of clinical careare to be cut back. If new monies are to be found, where will they come from - otherareas of government spending or perh aps from new taxes? Is the community willing topay extra tax? If outcome audit is so good for health, why not extend it to educationand to others areas of government spending? The cost implications are staggering. Sowhere do we go from here?

CONCLUSIONS

The publishing of card iac surgical mortality data is at once a blessing and a curse . Atleast in the case of the UK it was almo st certainly a political stunt more akin to spin than

information in response to an embarrassing pub lic scandal in Bristol (and Australianpol iticians may be tempted to follow after the Bundaberg scandal). The aim was to reassure a suspicious public rather than to trul y inform. The data, for the reasons outl ined

abo ve, is quite inadequate to inform patient decision making, and it is even possible thatthe 30 day mortality might be the wrong mortality end-point (Ada b et al. 2002; Edwards& Taylor 2003 ). In the socialised health system of the UK, where pat ient choice regarding

hospitals or doctors is virtually non-existent, no one could seriously claim report cardsimpro ve pat ient choice. The claim that the fall in cardiac surgical death s in UK was due

to report card s seems unl ikely to be true. While it is poss ible that report card s had aneffect, it seems to me that it wou ld have been marginal when compared with the impactof the NSFCHD programme, which led to improvements in a wide range of variableswhich are known to strongly impact cardi ac outcome. The effect of individua l variableshas not been analy sed so one cannot be categorical.

In healthcare systems with a significant private sector such as the USA, analysis ofthe hard data does not support a conclusion that report cards improve clinical outcomes.It is possible, indeed probable, that they foster injust ice by allowing the rich and socially

empowered to use market forces to pick and choose surgeons, leaving the poor and disempowered (who have the worst health ) to the worst surgeons, or to no surgeon at all,if we are to believe the data about defensive surgery.


So much for the curse, what of the blessing? The entire debate surrounding reportcards has opened a Pandora's box in terms .of audit and patient-focused information.This may be their lasting legacy. Do not assume doctors have been working away insecret. Far from it. Many doctors are actively engaged in serious audit as individuals or

groups along the lines of my ICU audit. Others, such as intensivists, oncologists, surgeonsand interventional cardiologists contribute to large multi-centre databases which can beused to set standards of care and to allow comparison of one doctors' results againstcolleagues. Numerous guidelines exist to help doctors achieve current best practice. Theseare all in the public domain, but as they are meant for health professionals, the information is not readily accessible or intelligible to the public.

The report card issue should be used as a starting point for a debate which asksquestions about what information doctors need to judge their own performance; whatinformation government, health insurers and the wider community need to be assured

that doctors are working to highest clinical standards while at the same time avoidingprofligacy; what information patients need and in what format to allow them to makeproperly informed choices about their healthcare; how is this to be funded, and if, as islikely, funds are insufficient, where do we draw the line between information and clinical

care at the bedside when both are competing for the same funds? Presumably, the ac

countability debate will move beyond doctors to other health carers. It must definitelyembrace health managers and bureaucrats. They come under intense examination whenit comes to budgetary and bureaucratic targets, but they sometimes seem blind to clinicalgovernance (patient safety, clinical standards). The tragedies of Bristol, Bundaberg andStafford were gross failures of clinical governance as much as examples of clinical incompetence, and if we are to learn anything from them, it is that corrupt managers can be a

far greater hazard to public safety than incompetent doctors.Lastly, who should oversee this information-accountability revolution? To allow

government this role (or to have government legislate this role for itself) would be amistake, as it imposes a very real risk of data manipulation for political purposes (Pitches2003). As far as doctors are concerned I would prefer audit to be conducted by the individual doctors, but under the auspices of the appropriate College or professional body,who would conduct random audits to ensure honesty and accuracy. But then, it mightbe asked, who polices the policeman? Somewhere we need trust. The community needsto inform the debate as to the type of information required from its doctors. The greataccountability debate is just starting. In our rush to know more and measure more weneed to remember that good quality information which has high utility is a very expensive

20.17 SURGICAL REPORT CARDS ARTlClfS

commodity, and it is important to leave enough funds, doctors and nurses to look afterpatients.

REFERENCESAbdul-Hamid, A.R. & Mulley, G.P. 1999 . Why do so few older people with aortic stenosis have

valve replacement surgery? Age and Ageing 28: 261-64.Adab, P., et al. 2002 . Performance league tables: The NHS deserves better. British Medical Journal

324 : 95-8.Anderson, T. & Hart, G. 1988. An overview of Australian and New Zealand Critical Care Resources.

In ANZICS Intensive Care Survey. Carlton: ANZICS Research Centre .Australian Institute of Health and Welfare 2007. Labour force: Medical. Australian Government

Publication. Available at : www.aihw.gov.au/labourforce/medical.cfm.Bonow, R.O. 2005. The Cardiovascular State of the Union . Circulation 111: 1205-07.Boseley,S., 2009. 'Heart operation death rates fall after data published'. Guardian (UK). Available

at : www.guardian.co.uk, 29 July, 2009 .Bouma, B.]. et al. 2001. Variability in treatment advice for elderly patients with aortic stenosis : A

nationwide survey in the Netherlands. Heart 85: 196-201.Canadian Health Services Research Foundation 2006. Myth : People use health system report cards

to make decisions about their health. Mythbusters. Available at :http://www.chsrf.ca/mythbusters/htmllmyth23_e.php.

Care Quality Commission 2008 . Heart Surgery in the UK: Survival Rates.http:/heartsurgery.cqc;org.uklSurvival.aspx.

Chambers, J. 2005. Aortic stenos is (editorial) . British Medical Journal 330: 801-02.Clarke , R. 1996. Outcome as a function of annual coronary bypass graft volume. Annals ofThoracic

Surgery 61: 21-26.Davies, G. 2005. Queensland Public Hospitals Commission of Inquiry. Final report available at:

http ://www.qphci .qld.gov.au .Department of Health . 2004 . The National ServiceFramework for Coronary Heart Disease: Winning

the War on Heart Disease: Progress Report.Department of Health . 2008. The Coronary Heart Disease National Service Framework: Progress

Report.Dranove, D., et al. 2003 . Is more information better ? The effects of report cards on health care

pro viders. Journal of Political Economy 111: 555-85.Edwards, M-B. & Taylor, K.M. 2003 . Is 30-day mortality an adequate outcome statist ic for patients

considering heart valve replacement? Annals of Thoracic Surgery 76: 482-85.EuroSCORE: http:/www.euroscore.orglcalc .html.Fung, C. et al. 2008 . Systematic review: The evidence that publishing patient care performance data

improves quality of care. Annals of Internal Medicine 148 : 111-23.

SURGICALREPORT CARDS ARTICLES 20.18

Henderson, A. et al. 1997. Prospective stud y of outcome and cost in a major Australian IntensiveCare Unit using the APACHE III severity of illness scoring and prediction tool. ClinicalInten sive Care 8: 58-62.

Kalavrouziotis, D. et al. 2009. The European System for Cardiac Operative Risk Evaluation(EuroSCORE) is not appropriate for withholding surgery in high-risk patients with aorticstensis: a retrospective cohort study. journal of Cardiothoracic Surgery 4: 32.

Knaus, W.A. er al. 1985. APACHE II: A severity of disease classification system. Critical CareMed icine 10: 818-29.

Knaus, W.A. er al. 1991. APACHE III Prognostic System: A risk prediction of hospital mortalityfor critically ill hospitalized patients. Chest 100: 1619-36.

Kojodjo j, P. et al. 2008 Outcomes of elderly patients aged 80 and over with symptomatic, severeaortic stenosis : Impact of patient's choice of refusing aortic valve replacement on survival.Quarterly Journal of Medicine 101: 567-573.

Lindroos, M. et al. 1993. Prevalence of aort ic valveabnormalities in the elderly. An echocardiographystud y of random population sample. Journal of the American College of Cardiology 21:1220-25.

MacDonagh, O. 1958. The 19th Century revolution in government : A reappraisal. HistoricalJournal1: 52-67.

Marasco, S.F.; Ibrahim, J.E. 2007. Is the reporting of an individual surgeon's performance doingmore harm that good? In S. Clarke and] Oakley, (eds). Informed Consent and ClinicalAccountability, pp. 200-01. Cambridge: Cambridge University Press.

MORI Poll 2007. Royal College Physicians London, November.New York State Department of Health 2008. Adult Cardiac Surgery in New York State 2003-5.Nilsson,]. et al. 2006 . Comparison of 19 pre-operative risk stratification models in open-heart

surgery. European Heart Journal 27: 867-74.O'Neill, O. 2002. A Question of Trust. Cambridge: Cambridge University Press.Pitches, D. 2003 . How to make a silk purse from a sow's ear - a comprehensive review of strategies

to optimi se data for corrupt managers and incompetent clinicians . British Medical Journal

327:1436-39.Schneider, E.C. & Epstein, A.M. 1998. Use of public performance reports: A survey of patients and

surgery. Journal of the American Medical Association 279: 1638-42.Smith, R. 2009 . NHS targets may have led to 1200 deaths in Mid-Staffordshire. The Telegraph

(London), March 17. http ://www.telegraph.co.uk/health/healthnews.Sorell, T. 2007. Safety, accountability and choice after the Bristol Inquiry. In S. Clarke and]. Oakley,

eds. Informed Consent and Clinical Accountability, pp. 52-75. Cambridge: CambridgeUniversity Press.

Vahanian, A. et al. 2006 . Aortic stenosis. In]. Camm, T.E Luscher and S. Serruys, eds. The ESCTextbook of Cardiovascular Medicine, pp. 626-32. Oxford: Blackwell Publishers.

Waley, A. 1938. The Analects of Confucius. London : Allen and Unwin .Warrd, D.A., Cox, T.M., Frith, J.D., eds. 2003 . Oxford Textbook of Med icine, vol 2. Oxford:

Oxford University Press.


Waterworth, P.O. et al. 2008 . Factors which influence the cardiac surgeon's decision not to operateon patients referred for consideration of surgery. Journal of Cardiotboracic Surgery 3: 9.

Werner, R.M. et al. 2005 . Racial profiling: The unintended consequences of CABG report cards.Circulation 111: 1257-63.

Werner, R.M. & Asch, D.A. 2007. Examining the link between publicly reporting healthcare qualityand quality improvement. In S. Clarke and J.Oakley, eds. Informed Consent and ClinicalAccountability, pp. 213-25. Cambridge: Cambridge University Press.

Cite this articleas: Henderson, Alan. 'Surgical reportcards:The myth and the reality'. MonashBioethics Review 28 (3): pp. 20.1 to 20.20.001: 10.2104/mber0920.


Documents

Surgical Report Cards