EDUCATION & DEBATE

Opening the black box: an encounter in the corridors ofhealthservices research

Catherine Pope, Nicholas Mays

AbstractHealth services research has become more promi-nent as a result of the NHS reforms. Both providersand purchasers want to know exactly where themoney is spent and how it could be used moreeffectively. How best to obtain information abouthealth services is the subject of some debate withinand between disciplines engaged in such research.Because of their training doctors are often scepticalof anything other than formal clinical trials andresearch which produces statistical data. Somesociologists argue that another way to find out whatis actually happening in the NHS is to observepeople at work and talk to them. This article debatesthese differing views of research methods. Foreffective research both quantitative and qualitativeapproaches need to be used.

This paper presents a dialogue between two conflictingvoices from health services research. It is presentedprimarily to inform and stimulate debate and it there-fore adopts a style which is unusual in this journal. Thepolarisation of views, inherent in the structure of adialogue, may oversimplify complex issues at the heartof the debate, but we hope that it highlights severalimportant conflicts which remain unresolved.The setting for the dialogue is the corridor outside

the office of the director of a large and successful healthservices research unit. The director (who has animpressive record of quantitative research) meets arecently appointed sociologist....

SOCIOLOGIST: I'm glad I've caught you. It's about thisresearch proposal you've just turned down what doyou mean, "It's not proper health services research?"DIRECTOR: Well, you were going to look at only twohospitals. What sort of a sample is that? Why don't youtake up my earlier suggestion of doing a randomisedcontrolled trial?soc: Because it won't tell you what you need to know.My project was a reasonable attempt to find out what'sreally going on in those two hospitals.DIR: I'm sorry, but we have to convince the medicalresearch establishment that we can deliver high qualitywork not these small scale, unquantifiable studies ofyours. Clinicians often see health services research asthe soft option and easy to carry out.' We need to wintheir respect.soc: And how, exactly, are you going to do that?DIR: We've got to undertake good, credible, scientificresearch. Science is respected and understood byclinicians (after all it's the foundation of medicine).soc: Do you mean science in general or a particularimage of "hard" science like economics with all itsequations. To my mind, what I do as a medicalsociologist is just as scientific.

DIR: You're entitled to your view naturally, butclinicians won't understand what you do. The modelof science they know is an experimental one-therandomised controlled trial used to test drugs andsurgical procedures. We can test health services inexactly the same way. A fraction of current healthservices research in the United Kingdom consists ofrandomised trials'; we need many more-you knowthe sort of thing, classic trials like Mather's work inthe late 1960s23 which compared the treatment ofmyocardial infarction at home and in the hospitalcoronary care unit. There was a higher mortality after amonth in the group treated in hospital. A year laterthere was still a significant advantage for the patientswho went home.soc: Hang on a minute. Wasn't that the trial whereonly about a quarter of the patients were actuallyrandomised? It's hardly a celebration of the experi-mental method. It was fraught with problems.

"Methodologicalpluralism is vitalin an applied subject like health

services research."

DIR: Yes, but the study was repeated by another team.The second time the researchers randomised most ofthe patients and they showed no significant differencein mortality in the home and hospital groups at sixweeks.4 From these studies we've developed criteria toidentify who needs to go to a coronary care unit andwho doesn't.

Who applies the results oftrials?soc: But does anyone actually use those criteria?DIR: I don't know. I'm just a researcher-not acardiologist. It's not my job to implement research. AllI do is produce basic knowledge.soc: As far as I can see, your contribution to basicknowledge is well and truly ignored. Not just incoronary care-there are other examples. Numeroustrials have evaluated the various procedures performedduring pregnancy and labour (Iain Chalmers has evengone to the trouble of collating them) but very few ofthese ideas have changed obstetric practice.5DIR: I can't help it if some clinicians are cussed.Anyway, you can't dismiss the experimental methodjust because some irrational people choose not to putthe findings into practice. Randomised trials haveenormous potential for improving health policy-at amuch higher level than individual specialties likecoronary care or obstetrics. Take something asfundamental as the NHS and Community Care Act4-we could have tested whether general practitioner

BMJ VOLUME 306 30 JANUARY 1993

Department ofPublicHealth Medicine, UnitedMedical and DentalSchool, St Thomas'sHospital, London SEI 7EHCatherine Pope, researchassociate

Health and Health CareResearch Unit, Queen'sUniversity ofBelfast,Institute ofClinicalScience, Belfast BT12 6BJNicholas Mays, director

Correspondence to:Catherine Pope, HealthServices Research Unit,Department of PublicHealth and Policy, LondonSchool ofHygiene andTropical Medicine, LondonWC1E 7HT.

BMJ 1993;306:315-8

315

on 1 January 2021 by guest. Protected by copyright.

http://ww

w.bm

j.com/

BM

J: first published as 10.1136/bmj.306.6873.315 on 30 January 1993. D

ownloaded from

fundholding was better than health care purchasing bydistricts. We should have done something like theRAND health insurance experiment.'8soc: The what?DIR: The huge project in the United States whichrandomised people to different health insuranceschemes to look at the consequences, including theimpact on their health. That's the kind of work weshould be getting into here.soc: I'm sorry, but I still have real problems with thispicture of the experiment as an ideal. This is a modemhealth services research unit but you hold an anti-quated view of science. It only seems to include theexperimental model drawn from the natural sciences.I'm not even convinced that the natural sciencesactually work like that9 and I'm not sure you have anyright to assert that the randomised controlled trial isthe best method. It has its limitations.

Alternatives to experimentsDIR: I think you're just against experiments.soc: Not entirely, but your "one best method"argument reminds me of a very old debate in sociologyabout positivism ...DIR: Do you have to talk in "isms"? If you could put itin plain English I might be able to understand.soc: Let me draw an analogy, then. The differentmethods employed by social scientists are like thedifferent views of the surgeon and the epidemiologist.Surgeons leam through direct experience of individualcases-through what they see, hear, and feel at theirfingertips. In contrast the epidemiologist views thesurgeon's patients at the aggregate level as clusters ofvariables. Have you got that?DIR: Yes, but I've never given much credence toanecdotal evidence from surgeons. Go on.

soc: Well, between those two extremes there is a wholerange of theoretical perspectives and research methodsto choose from, both qualitative and quantitative.What I want, retuming to my analogy, is for thesurgeon's view to be given a place in the scheme ofresearch alongside the epidemiologist's. Methodologi-cal pluralism is vital in an applied subject like healthservices research. Even the Medical Research Councilrecognises that health services research "is typicallymultidisciplinary, bringing together as appropriate

Surgeons learn through what they feel at their fingertips, but epidemiologists view patients as clusters ofvariables

expertise in biological and clinical science, epidemi-ology, statistics, economics and the social sciences."'0If you only use experiments you're using a very limitedtool box.DIR: I wasn't arguing just for randomised controlledtrials-but we do need hard facts like those whichexperiments provide.soc: Yes, but you judge all facts using hard science asyour gold standard. The point is that some things inhealth services can't easily be looked at with quantita-tive methods alone. Qualitative methods could help bylooking at health care organisation and delivery-at theprocesses of care.

What is meant by process?DIR: But process is simply what health services do topatients. We're interested in the product of healthcare, the outcome, the results of intervention."I If thepatient dies it's a bad outcome and I know there'ssomething wrong with the process. End of story.

cc~~~~~~~~~~~ e

"Process is simply what healthservices do to patients. We're

interested in theproduct ofhealthcare, the outcome."

soc: That's oversimplifying the situation. We need awider definition of process. It's more than just whathappens to individual patients. It's also about organisa-tions and the people within them-not just the patientwho dies, but the doctors, nurses, auxiliaries, plan-ners, administrators, clerks, and porters, and thenoisy, chaotic interaction between them and thestructure that surrounds them. There is a black boxmarked process and we haven't even begun to open it.DIR: So what exactly would you do?soc: Well, for a start, I would open up our fullmethodological tool box and start using techniquesother than randomised trials and models of researchborrowed from epidemiology. Perhaps health servicesresearchers could begin to use some of the qualitativetechniques available.DIR: Aha! I knew it. You want us to conform to yourorthodoxy ...soc: No, mine isn't the only approach. All I'm askingis that you begin to take these methods seriously andconsider them alongside your own quantitative skills.After all, market researchers in the no nonsense worldof retailing and commerce often use qualitative andquantitative methods together.DIR: What exactly are these qualitative methods you'reoffering?soc: Well, what about observational studies, for astart?DIR: But we do lots of those. We've done lots ofcomparative work, case-control studies ...

soc: Oh dear! We're not even talking the samelanguage here. I didn't mean case-control studies. Imeant observation. You know, being there, looking,and listening. I was thinking of ethnography, whichmeans you have to immerse yourself in the situationand talk to the people involved like an anthropologistwould.'2 That's just one example of an approach whichgets away from counting events and controlling forextraneous variables. It's about trying to understandwhat is going on, almost through the eyes of theparticipants themselves.

BMJ VOLUME 306 30 JANUARY 1993316

on 1 January 2021 by guest. Protected by copyright.

http://ww

w.bm

j.com/

BM

J: first published as 10.1136/bmj.306.6873.315 on 30 January 1993. D

ownloaded from

Waiting lists can't be seen as bus queues

DIR: Sounds like an excuse to loaf around doingnothing in particular to me. What can ethnography tellus about the big issues? For instance, I bet it can't helpus solve the problem ofwaiting lists? Can your preciousethnography tell us anything which would be ofpractical use about managing these queues?

Value ofethnographysoc: Only that they're not queues. Isn't that worthknowing?DIR: What pretentious, counterintuitive rubbish. Wemight not know how best to manage waiting lists, butwe don't need sociologists to complicate the basics bytelling us they're not queues. They're great longqueues ofpeople waiting to go into hospital.soc: No, they're not. By saying they're like busqueues, you've made lots of assumptions. If you reallywant to understand a waiting list you need to get inthere are find out how it is organised and managed. Thebest way of doing this is to study the people whoactively assemble and maintain the waiting lists. Thenyou see that waiting lists seldom resemble anything likethe formal queue which operations researchers are sofond of modelling.DIR: I'm still puzzled about how you got this idea.soc: By studying one district in detail using theethnographic methods I described." By observing howa list is managed I found out that although lists are keptchronologically, patients seldom come off the list inthat order. The office staff and the surgeons used thelist as a pool of work they would dip into-indeed asurgeon might deliberately choose a recent addition tothe list over someone who had waited far longer on thegrounds ofgreater urgency. . .

DIR: And quite right too.soc: ... or simply because they remembered thepatient. There were all sorts of other processes thatworked against the idea of a simple queue whichmanagers needed to know about.DIR: I take your point, but what about wider, inter-national debates? What about explaining variations inthe rates of common surgical procedures like chole-cystectomy and hysterectomy between regionsand countries. Quantitative work by people likeMcPherson, Wennberg, and so on"' can tell us aboutthat variation.soc: And I suppose you'd like more of the same so youcan go on pinpointing variation and replicate the

studies which have been done to show the same thing indifferent places, or maybe to include a few moreexplanatory variables in your statistical model?DIR: Well, yes...soc: But clinical variation raises other questions whichneed to be answered. What we really need to do now isstart uncovering how those rates are generated by theactions of individual clinicians. Take something likeWennberg's concept of the surgical signature, used todescribe the different profile of surgical work per-formed by different surgeons." What we need to knowis how those "signatures" get written. And this gets usback to looking at process. We need to know thesequence of events which take place before the pattemsof surgical variation are produced.DIR: So what do you think your approach can offer?soc: For one thing, it could tell us more about howvariation is constructed. Mick Bloor's qualitative workon adenotonsillectomy is a perfect example of the kindof study I'm talking about.'6 He carried out anobservational study of ear, nose, and throat outpatientclinics and showed that there were systematic varia-tions in patient assessment routines among consult-ants, rooted in differences between the specialists intheir informal decision making rules. If you combinesuch work with quantitative data you can begin toexplain how variation occurs.

Outsider's view ofsociologyDIR: Your programme for looking at process is all verywell, but this is exactly what your lot, medical sociolo-gists, have ignored. '7 Medical sociology has long sincegiven up looking at process-it's too busy experiencingillness and waffling on about doctor-patient interaction.soc: Perhaps, but part of the reason lies in the cultureof health services research. In the United Kingdom it'sdriven by medicine and there aren't many posts forsocial scientists.'8 You only have to look at what getsfunded and who evaluates the proposals. There's verylittle room for the qualitative work I've been talkingabout: if it is there it tends to get tacked on to anexisting project when the sociologist is brought in toprovide expertise on survey design or interviewing orto use a standard measure of patient "quality of life."DIR: You can't blame me for your failure to securefunding. Anyway isn't the Medical Research Councilcanvassing medical sociologists for grant applica-tions?"9

"Waiting lists seldom resembleanything like theformal queue."

soc: Yes, but they mostly seem to have people like youassessing the proposals. It's no good having people whoknow nothing about qualitative research applying theiryardsticks of experimental science to all types ofresearch.DIR: Well then, I certainly can't argue your case foryou. I only know about my approach.soc: But you could back my project?DIR: The decision's made. But I'll tell you what we'lldo. Come back to me in a few weeks with anotherresearch proposal. After today's discussion I should bea bit better at understanding what you're driving at!soc: That's something, I suppose.DIR: Could I make one last suggestion? Your researchproposal wasn't very user friendly. You could do worsethan take a leaf out of the health economists' book.

BMJ VOLUME 306 30 JANUARY 1993 317

on 1 January 2021 by guest. Protected by copyright.

http://ww

w.bm

j.com/

BM

J: first published as 10.1136/bmj.306.6873.315 on 30 January 1993. D

ownloaded from

When I started out, nobody had heard of healtheconomics; now every provider unit in the healthservice wants one. People seem to want health econo-mists, up to a point, and even epidemiologists becausethey boast a set of tools to offer managers and doctorsfor opening what you called the black box. Theeconomists didn't get to this position by hanging backand wingeing from the sidelines. If, as you claim,medical sociology, and your ethnographic methods,can really open up this realm of process and tell us whatis going on in the "black box" then you've got to bemore entrepreneurial. Change your name to Pandorawhile you're at it, people might be less inclined to bedismissive!

1 Fowkes FGR, Garraway WM, Sheehy CK. The quality of health servicesresearch in medical practice in the United Kingdom. BMJ 1991;45:102-6.

2 Mather HG, Pearson NG, Read KLQ, Shaw DB, Steed GR, Thome MG, et al.Acute myocardial infarction: home and hospital treatment. BMJ 197 1;iii:334-8.

3 Mather HG, Morgan DC, Pearson NG, Read KLQ, Shaw DB, Steed GR, et al.Myocardial infarction: a comparison between home and hospital care forpatients. BMJ 1976;i:925-9.

4 Hill JD, Hampton JR, Mitchell JRA. A randomised trial of home versushospital management for patients with suspected myocardial infarction.Lancet 1978;i:837-41.

5 Chalmers I, Enkin M, Keise MJNC. Effective care in pregnancy and childbirth.Oxford: Oxford University Press, 1989.

6 National Health Service and Community Care Act 1990. London: HMSO, 1990.7 Ware JE, Brook RH, Rogers WH, Keeler EB, Davies AR, Sherboume CD,

et al. Comparison of health outcomes at a health maintenance organisationwith those of fee-for-service care. Lancet 1986;i: 1017-22.

8 Welch BL, Hay rW, Miller DS, Olsen RJ, Rippey RM, Welch AS. The Randhealth insurance study: a summary critique. Med Care 1987;25:148-56.

9 Bhasker R. The possibility of naturalism: a philosophical critique of the contemp-oraty human sciences. Sussex: Harvester Press, 1979.

10 Medical Research Council. Corporate strategy, 1989. London: MRC, 1989:48.11 Cochrane AL. Effectiveness and efficiency: random reflectionts on health services.

London: Nuffield Provincial Hospitals Trust, 1972.12 Hammersley M, Atkinson P. Ethnography: pnrnciples in practice. London:

Tavistock, 1983.13 Pope C. Trouble in store: some thoughts on the management of waiting lists.

Sociology ofHealth and Illness 1991;13:193-212.14 McPherson K, Wennberg JE, Hovind 0, Clifford P. Small area variations in

the use of common surgical procedures: an international comparison ofNewEngland, England and Norway. NEnglJMed 1982;307:1310-4.

15 Wennberg J, Gittelsohn A. Variations in medical care among small areas.ScientificAmerican 1982;246:100-1 1.

16 Bloor M. Bishop Berkeley and the adenotonsillectomy enigma: an explorationof variation in the social construction of medical disposals. Sociology1976;lO:43-61.

17 Hunter D. Organizing and managing health care: a challenge for medicalsociology. In: Cunningham-Burley S, McKeganey NP, eds. Readings inmedical sociology. London: Tavistock/Routledge, 1990:213-36.

18 Clarke M, Kurinczuk JJ, for Committee of Heads of Academic Departmentsof Public Health Medicine. Health services research: a case of need or specialpleading? BMJ 1992;304:1675-6.

19 Peatfield AC. The Medical Research Council, health services research andsocial science. Medical Sociology News 1992;17(2):15.

(Accepted 17 November 1992)

Authors have rights too

Michael E Dewey

Department ofPsychiatry,University ofLiverpool,PO Box 147, LiverpoolL69 3BXMichael E Dewey, lecturer inpsychological statistics

BMJ 1993;306:318-20

Without publication research can be of little value.When researchers approach publication there isample published guidance for them on what theirobligations are, and there are well known styleguides within each scientific discipline including, inmedicine, the Vancouver style. This article gives aseries of anonymous examples to suggest that theimpact of similar guides for editors has been patchyand to make some suggestions for better communi-cation.

The Vancouver style, "Uniform requirements formanuscripts submitted to biomedical journals,"' setsdown only the obligations of authors. The wideacceptance of this style guide, and similar ones in otherdisciplines, suggests that it fills a need. So far theredoes not seem to be an equally well known guide on theresponsibility of editors to authors and to referees.

Below, I illustrate some of the problems authorsexperience that could be avoided by editors followingguidelines. The examples do not identify the article orjournal concerned, but each example has happenedto me or my coauthors during submissions to whatare generally regarded as quality journals. Most ofthe examples are from Britain but European and theAmerican journals also figure. The order of the pointsin the article corresponds to the progress of an articlefrom submission to eventual publication not to per-ceived seriousness.

What iftwo papers with similar content arrive?There may seem to be no problem for an editor

under these circumstances: each paper is assessed onits merits and published accordingly. I submitted anarticle which caused disagreement between referees,and after a third opinion was sought it was rejected. Afew months later the journal published an articlecovering similar ground. The published paper wasmore extensive and a much better article, but an authordoes not have to be paranoid to wonder what went on.

Perhaps editors need to bear in mind what authors maythink when this sort of thing happens and keep thembetter informed.

What ifthe editor is also an author?If there are few good joumals in a specialty, editors

may not be able to publish during tenure unless somemechanism can be found to allow for this eventuality. Isubmitted an article to the joumal of which one of mycoauthors was editor. The rules of the organisationwhich owned the joumal outlined a procedure to befollowed that used a guest editor. However, thisprocedure was not explicit to the readership, and onlyby adding an acknowledgement to the article could wemake clear that the article had not been accepted justbecause the editor was an author.Banning the editor from publication in the joumal

seems extreme if there are few altemative outlets.Logically, the editor's research team would also have tobe banned, which would probably discourage potentialeditors even more. Whatever the procedure for dealingwith the problem the joumal should make it explicit.

How long should the author wait?When they acknowledge receipt of an article

joumals sometimes state how long authors shouldexpect to wait before receiving a decision, althoughfew make this information more widely available. Iwaited two years for the first substantive response toone article despite reminder letters to the editor. (Itwas then rejected, which added insult to the injury,although it was then accepted by another joumal).Another joumal has taken a year to respond on morethan one occasion.Joumals usually blame slow referees, but if they

have not replied within three months are they likely toreply at all? It is no real answer to say that authors couldwithdraw the article and resubmit elsewhere, as thechosen journal may be the most appropriate one.

318 BMJ VOLUME 306 3OJANUARY 1993

on 1 January 2021 by guest. Protected by copyright.

http://ww

w.bm

j.com/

BM

J: first published as 10.1136/bmj.306.6873.315 on 30 January 1993. D

ownloaded from