\

164 066

MITBORTITLI

--INSTITUTION

REPORT NOPUB DATE'MOTE

--kVAILA-8141-F-RO,

EDRS,..PRICEDESCRIPTORS

ABSTRACTIn this paper the author assesses 'the

state-of-the-art on quality assessment and monitoring or medical cmreand makes recommendations for needed research in this area. Following'a brief iutebduction, the content is presented in two sections. Thefirst, providing a frame'of reference, covets definitions; qualityassessment and program evaluation; relationship of quality.aadquantity; relationship of quality and cost; strategies of care;structure, process, and outcome; monitoring versus research; and theuses,,of outcoges. The second section presents.a catalog of neededresearch on assessing and monitoring the qualityNof medical care. Theresearch areas covered are as follows: basic explorations and studiesof what constitutes quality, description of prevalent patterns and'sttategies of care, the epidemiology of quality, the relationshiP Ofstructure to process or outcome, development of basic tools forassessment, specification and testing'of system-design elements,comparative studies of quality using different approaches, furtherdevelqpment of promising current approaches, integrative measures oqualitle appliehtions,to special areas, consumer perspectives and th-econsumeWs role, quality assessment and monitoring, as a socialprocess, and effectiveness and the factors that influence it. Anextensive bibliography is attached. (EM)

-t

DOCONIVT IMMIX

CR 019 010

DonabSdian, AuntieNeeded Research in the Assessment and Monitoring orthe Quality of Medical Care. NCHSR Reseatth ItePolltSeries. . -

National Center for Health Services Research and

Detelopment (DHEN/PHS), Rockville, Md.DHEN-PHS-78-3219Jul 7846p.National Center for Health Services Researcit, officeof Scientific and Technics': Infor 0.°11, 3700.-East-Neat Highway,- R m 7$4, Hyatt vine, Maryland

20782- f-

Oft

817-$0.83 HC-$2.06:Plus Postage..

Bibliographies; -Evaluation CriteriavNeasurementTechniques; *Medics). Care Evaluation; MedicalServices; Models; Program Evaluation;-*4militr.

'Control; Research-Methodology; *Research Needs;andards; State of the Art Reviews '

S.

..

i s-

**********4************************************************************* Reproductions supplied by EDRS are the best that an be made *

* .from the original document. 4

4***********************4**************************** ******************

RESEARCH REPORTSERIES .

NeededResearchin the ,

Assessmentand

"Monitoringof the Quality.of MedicalCare

Avedis Donabedian, M.D., M.P.H.Department ofMedical Care 'Organization.School of Public HealthThe University of Michigan,Ann Arbor

July 1978

r

ft.

U.S. DEPARTMENT.OF HEALTH, EDUCATION, AND WELFAREPublic Health ServiceNational Center for Health Services Research

DHEW Publication No. (PHS) 78-3219 2

f

A

ts,

U.S. DEPARTMENT OF HEALTH.EDUCATION A WELFARENATIQNAL INSTITUTE OF

!EDUCATION

THIS bonmENT HAS -SEEN REPRO-'MAE!) ()wry( AS tECEIve0 FROMTHE PERSON OR ORGAPIATION ORIGINATING IT POINTS OF yIEW OR OPINIONSSTATED 00 NOT NECESSARILY REPRE-SENT OFFICIAL NATIONAL INSTI TUTE OFEDUCATION POSITION OR POLICY

National Center for Health Services Research

Research Report Series

The ReiearchReport Series is published by the Na-tional center, for Health. Services Research-(NCHSR) to provide significant research reports'in their entirety. Research keports are developedpy the principal investigators who conduct the re-search, and are directed to selected users of healthservices research as part of a continuing NCHSReffort to expedite the disseMination of rkew kno I-edge resulting from its project support.

Abstract

The purpose f this paper is to review, evaluate.

critically, an synthesize the literature oh qualityassessment nd assurance, including the appro-priateness f use of service, in order to arrive at a'cogent, d umented, and authoritative assessmentof. the s te-of-the-art. In addition to addressingquality ssessment as a research tool and qualityassura ce as an administrative tool, an attempt is

- blade to prbvide an understanding of the

slcpi

etiology 'of q -as a prerequisite to the de-s' medical c rams and systems. Majorcomponents of qtranitY:which are discussed in-clude: definitions; quality assessment and progradievaluation; relationshito,of quality and 'quantity;relationship of quality and c t; strategies of care;

. structure, prixess, and wilt ome; monitoring ,:wer-sus research; and the use bf outcomes. Rcom-mendations for further reseaF.ch in the assessmentand monitoring of the quality Vif medical care are

.'presented.I

'yr

11 This NCHSR Research Report was written by AyedisDonabedian, M.D., M.P.H., Professor of .MedicalCaie Organization, Department of Medical CareOrganization, School of Public Health, The Uni-versity of Michigan, Ann Arbor. This report, anextended version of a paper prepared for the De-partment of Medicine and,Surgery of the Veterans

1 Admini tration,. is one product of the work cue-rently eing supported by the National Center forHea .Services Research under grant number HS020 J. The final report from the work supportedby NCHSR is not expected to be completed til

June 1979..Dr. Donabedian was also supporte by, The University of Michigan, through sabbatical

leave, during part of this study.

Additional copiesof this report may be obtainedon. request from the NCHSR Office of Scientific

Ad Technical Information, 3700 East-Westighway, room 7-44, Hyvasville, MD 20782 (tel:17436-8970). Other current NCHSR publica-

i' tTons are announced on the last .pages of thispublication. , .

C

J

Prior to 1965, the year in which Medicare becamea'reality, efforts to improve the quality of medicalcare were involved mainly with the self-regulationactivities of the medical profession, althoughsporadic research had been conducted -in the areasof medical care process and patient outcome as farback as the mid-nineteenth century. Utilization re-view was emphasized with the passage of Medicare

c 1 portended a new awareness of public interest.1111 the quality of care. Dr. Lionabedian had the bp-

rtunity during the period of 1965-1967 to re-view and assess the state-of-the-art of quality as-sessment methodology and responded with severalpublications which have become classics in thefield,. Since that time, there has been a period ofsignificant, if not remarkable, growth in the bodyof knowledge encompassing' quality -assessmentand assurance. A notable development in this areawas the implementation of the Experimental Med-ical Care Review Organizatibn (EMCRO) programin 171 by the National Center for,Mialth ServicesResearch. This program esta ed the model fol.,lowed in the development f the ProfessionalStar Bards Review Organization (PSRO) pr9gramand encountered many of the problems sub-sequently experienced in that program. TheEMCRO program-addressed problems of criteriadevelopment aid evaluation, organizational eat-terns, development of assessment and assurancetechniques, impact evaluation, an&many of theother emerging issues.of the day.

In October, 1972, Pu)blic Law 92-603 establishedformal PSRO review of medical services reim-bursed under th Social Security Act, and interestin all facets of quality research received added im-petus. The PSRO legislation has the potential forprofound impact on the cost and quality of medi-cal services and the form of health services deliv-ery; however, serious concern- has been voicedconcerning the wisdom of its current mode of im-plementation, aspects of which have not beenrigorously validated. -. °

In the past decade, a considerable body of HI

knowledge has been gathered which requiresthoughtful review, evaluation, and synthesis inorder, to assess the present state-of-the-art and toallow meaningful projections of further researchstrategies. It is in this framework that Dr. Dotiabe-dian suggests that "it is necessary from timetime to pause and take stock of what has beendone, so that it may be clearly understood andfuture effort redirected." The synthesis which he,provides is intended to mold together the re-search, operational, and policy concerns of thehealth establishment with regard to the quality ofmedical care at a time when major changes in thefinancing and organization of health services areon the horizon.

Gerald Rosenthal, Ph.D.DirectorNational CenterServices Research

July 1978

for Health

.0"

#11°REWORDvii NTRODUCTION

1 FRAME OF REFERENCE41Definitions

EvaluationQuality Assessment and Program Evaluation2 Quality and Quantity.

Quality and Cost3 Strategies of Care4 Structure, Process, Outcome5 Monitoring Versus Research

The Uses of Outcomes

PROPOSALS FOR RESEARCH

Basic Explorations and Studies of What Constitutes' QualitySpecification and Assessment by Modeling

9 Empirical Testing of Strategies of CareIncrements of Cost and of Quality

10 Studies of the Client-Practitioner RelationshipViews of What Constitutes QualityContinuity, and Coordlnailon as Attributes of Quality

11 Description of Prevalent Patterns and Strategies of CareStudies of thd Elements of Clinical BehaviorIdentification of Styles and Strategies

12 Comparisons of Norms with PracticeHomogeneity and Heterogeneity of Performance

The Epidemiology of Quality13 The Epidemiology of Quality Among Providers

The Epidemiology of Quality Among Clients

14 The Relationship of Structure to Process or Outcome

Development of Basic Tools for AssessmentSpecification and Measurement of Outcomes

16 Improvements in the Medical RecordThe Criteria and Standards of Quality

17 MOnetiry Measures of Costs and Benefits.

16 Spilcification and Testing of System -Desiga Elen ntsSpecifying the:Appropriate Cut-Off Pointfin Sta ardsSampling and "Enrichment:' Technique,

19 Larger Elements of Design 1-

dompirative Studies of Quality Using Different Approaches\

Furthei Development of Promising Current ApproachesOutcomes as Measures of Quality -

%Qtitcomes.as Cues. and Motivators for Process Assessment) C..---\

a

t,

vi 20 The Otcurrence of Preventable Adverse EventsThe Occurrence of Preventable Progression di illness or Disability

21 Second Surgical Opinion ProgramsComputer -Aided Management

...Integrative Measures of Quality.

Applications to Special Areas

22 COnsumer Perspectives and the Consumer's RoleRepresenting Client Values in the Definition of Quality

Clients as Sources of InformationParticipation in Monitoring

23 Membership in Auditand Utilization Review Committees

Patient Contributions to Care

Quality Assessment and Monitoring as a Social Process

24 Effectiveness and the Factors That Influence It

25 Changes in Physician and Client BehaviorEffect of Technical Design Characteristicsintra-Institutional "Social Design" Characteristics

28 Supra-Institutional Characteristics and Medical Technology

27 Social-Organizational Characteristics and Medical Technology

Case Studio of Successful and Unsuccessful ProgramijitiVEffeciveness and Cost-Benefit Studies

28 EPILOGUE \

29 References,36 Curreit NCHSSI-Ptiblications

-114,F

Not too' long ago, the quality of medical care wasa matter almost exclus ely in the professionaldomain. Any introduction of the subject in a morepublic setting was to be done, if at .ell, gingerly,almost apologetically, surrounded liy a cloud ofcaution designed to appease the wrathful and con-temptuous professional. How times have changed!"Quality" is now a term perhaps too easily bandittdabout; and there is little hesitance in proposingthat quality can be measured, or that itican be en-forced as a matter of public and administrativepolicy. But this mood of almost belligerent confi-dence is perhaps premature, for there is muchabout the concept of quality that is elusive. unde-fined and unmeasured. Our, knowledge- of_how 10

go about assuring quality is equally frail.The academician who seems to he pleading l or

inaction while proposing further research is astock figure of ridicule in our gallery of publicfools. This is a role'l shall tryio avoid. My thesii isthat while some of us go about doing the best theycan with. what is known, the effort to examinecritically what is being done, and to find ew and rbetter ways of doing it, should not be rel ed.

This pa r is offered as a modest contribution4,in this fu her. exploration. It will present a catalogof needed research that is sufficientlyotganized toavoid, being a mere haphazard listing of things.But first, a general framework is needed thatexplains and justifies the choice of research topicsand their organization into a classification. Theframework must also indicate what subjects areexcluded from consideration. For if reasonablystrict limits are not set, at the very beginning, theconcept of quality has a tendency to expand, sothat it embraces every evaluative statement aboutany aspect of the health care system whatever.

4

:mite claim that the concept of quality is enrichedben this happens. In my opinion!, this concept

ca ) also become attenuated and less useful.Whether I ant right or wrong, it is clear that I mustcircumscribe my subject rather narrowly if' I am tocomplete this paper. For the same reason, thedelineation of the frame of reference must also be

1, sketchy, for a thorough specification of theframework is a formidable undertaking' in itself.All that is necessary. for now, is4kihat we establishsome common ground upon which to build.

vii

Jr

A frame of reforms*

Definitions

In some ways, my definition of quality will betar sually narrow. Quality is taken 0) signify a,judgment on a rather limited segment of the per-formance of some professional personnel. Primar-ily, our concern is with the services that are re-ceived and consumed personally and directly byindividuals, and with those profess ale who pro-vide such services. Thus, for instant environ-mental services and admin'strators f personalhealth services arc clearly tsidr ou scope; butthe patient -care service o physician. , dentists.pharmacists, nurses, socia workers, . rid otherpractitioners are clearly included. I belit e that itis also reasdnable to include the services o es-sionals, such as pathologists and .diagnosttradiologists, who generate the data that are usedin direct patient care.

Traditionally, the activities of these several pro-fessionals are seen ws se arate contributions.Hence, one speaks of the q lily of physician care,nursing care, social work, a so on. In this paper,I shall fo is traditit by focusing almost ex-clusively on e services of physicians, but this isnot by pref7r n9e. It is a necessary concession, inpart, to the reiStively less deNelsoped state of qual-ity assessment outside medicineybut, mainly, it is areflection of my ignorance of much of the work inassessing the quality of performance in the otherprofessions. In fact, an essential component of the

.toncept of quality is the interrelatedness of the',contributions of the several professionals in themanagement of .'a patient', illness and health. Amajor- item on the research agenda is, therefore,the development of "integrative" measfltres.of qual-ify that take into account this interrelatedness.

Another way inwhich the concept of quality canbe expande4 is by. acCeptirig a definition of healththat is -considerably "broader than the traditional

- emphasis on physical-physiolOgical function. Ishall includeipsychological and social well-being asnecesiary components of health, but only toextent that responsibility for them can bepriately undertaken Wry Fakysician, und

I4

sonably liberil definition of his r )1e. It is not use-ful to define he responsibility f a health carepractitioner, fir. any aspect ASt h th, so that itgoes b.eyond hi)X)uthilly and p tfOssionally§a nut ioned role, oet heInstrumentalities actually orpotentially urger his citntrol.

Duality assessment and program evaluation

Program evAluation differs from quality a3se.ts-64

mem by being tnore inclusive: It deals Jvith ac-tivities of health practitioners in addition to directpatient care. It also includes the performance ofprofessionals and nonprofessionals who provide

).no dlr t patient care. As a result, there is more tom evaluation than 'quality assessment; al-

though quality assessment is part of programevaluation, and in many instalrces its most imBsor-tant part.

In addition to this difference in extent or in-clusivity, there are differences that flow from thelevel of aggregation at which the assessment isperformed. -Program evaluation deals at theaggregate level with the manner in which a com-plex formal organization functions, and whether itmeets its socially legitimate goals or some specifiedneeds or wants of its clients. In so far as the healthcare function is concerned, program 'evaluationplaces greker emphasis on access to care, on otheraspects of retionrce allocation, on cost and effi-ciency,- Ind on ,overall impact on a clientele, acommunity or .a population. By contrast, it hasbeen customary for quality assessment to focus onthe appropriateness of care at the level of personalinteraction between' individual patients and prac7titioners, as judged by the proper application ofscience .to the needs that the patient and thepractitioner have jointly defined. However, ashealth care becomes more formally organized, andits financing collectivized; there is increasing pres-sure to introduce into the assessment of quality atthe 'individual level concerns that were previouslyconfined to the collective level. of analysis andassessment, This ha's raised serious ethical prob-lems for the practitioner who is now caught be-

9

41.

I

tweet' the iwo milltonei Cif reoptinsibility Inwards..V.I.1111. patient Iimi the toilet tivits. It hasthe huh I I

alto blurred lrmerls t !eat er distini trans betweenprogram evaluation and gustily arsrsrnlrnt filAthave now become outtally different es in emphasisand in the detail* of humiliation, Some unin&Hurts of this blurring mid overlap will be t !milted

Subiequelit sections otthis paprt.

Quality and quantity

e overlap riWeeli iliiaTifiTV and quaint-- is title

con it through whit Ii plow ant evaluation and2 clu,tlie Assessilicill flow into each other, obvious's,

quantitative adequat s is .1 net resat 5' pi et onditionfor quality tate. This means that access tot ale and

insufficient Tries.of servile are legitiniate elementsin atisesking the of ( .11 lilionel II Ans-

:11 well as of tin' ()grant .1% A u 'Waif

11(1(111 A« C'5% .10)11 use heed mtitiiiiiingattention het use so mans plow ams air obss-

-,sivelt pivot c.upied. %till' I uring utilization andcutting costs, even though they mat pas lip serviceto the important r of eliminating insuffit ient i arr.

Excessive ((Sr of servile usually pool (Iti:11-

lit for a valaik of ski:titans todire(

(*it Of reasons. For one, it implies ain the «intim I of patir care: all in-

t tweed from step to step along the. mostth, selecting the pre( est's necessary and

sufficient procedures to arrive at a diagnosis an k.7

to institute treatment. This model of logical andparsimonious progression has traditionally "wen

the hallmark of virtuositt in medical care, andmans still hold it as the ideal, although much ofcurrent acceptable practice MAN have departedfrom it. Note, how-ever, that this is an ideal ofminimum redundancy, but not necessarily ofminimum cost. It 'is possible, for example, thatusing current technology, a strategy of extensive,almost indiscriminate, initial testing will lead to an.earlier diagnosis, With lower likelihood of error. atlower average cost, at least in some situations. 1

so, the criterion of lowest cost could 1w in Conniffwith the criterion of least redundancy as elementsin the definition .of quality. We shall return to this,.

interesting confrontation.\- In other ways*, the identification of poor quality

With excessive use is less problematic. Medicalprocedures, though intended to be beneficial, arenot without risk which, in some cases, is consider-.able. It is reasonable to assert that the concept ofquality includes the criterion that in no case should

the benefits expected from any procedure he lessthan the risks it poses. If so, a problem in assess-mem is the measurement of risks and benefits in_amanner (fiat permits a comparison.

Still another undesirable consequence of exces-sive use is that by allowing some to have too much,there is'less available to others. There is an obvious

Ion nil 1116411104 alum of Irmill11 14, W lit il is poorquality at the aggiegmlfr level. It I% unlot nog toare that, in this 11141Alitr, fir ifrilii of 'Callicillimisiliir at liie level of i ulividual patient Late.11111 at lilt, 4 (r( tivr level I all Ito thr same be-ha V 14U , train atiog ti fiatillisfliolIN 4 iniaelt e ofquaint assritsm4:and plow ant evaluation,

()tn. esittoi le that ant evaluative statementabout theTplatility of tare:beyond a mete deicriplion of it. is a judgment about some aye( 1 of goal

Its. Flu that 1 eason,..mtlitation teview and qualityassessment at e iiirxti ii .11)15 intet twined. mid willLou' so) 4 ottsitlted in this paper.

Quality and coat

We have alreadt had intimations that t ost is in)pin algid Fn the cow pt of quaint in a manner thatis likely to (mist us problems. I am not referringsimply iv the fat t that the mpit it al relationshipsbetween quality and cost ate essentiallyexplored. I Ain (.1 MCC r ut' 'thy conceptualtoonctions in the defiition of quality itself'.

The basic onsiderations that link quality to costsare essentially the same that connect quaint toquantity: monetary cost, benefit to health, and riskto health.

There are two ways in whit h the monetary costsof inputs into care can increase without any in-crease in the quaint of care. First, the elements ofservice that go into patient care can be providedineffitiently:l'flus, hospital care will l costlierhas .necessary T hitspital beds are emptv., or thehospital is improperly managed: or if phi sicians

do the work of nurses and the latter the work ofAndes. Second, the elements of care it be com-bined and sequenced in a manner thai does notrealize their NI) potential to improve health.

The fundamental attribute of these deficienciesin the organization, production and application ofcare.is that there are added costs withoift eitheradded benefits or added risks. Can this be con-strued. as poor quality of care?* I have already ar-gued that at the collective level of analysis the an-swer could legitimately be in the affirmative be-cause, when resources aft scarce, wasteful produc-tion and implementation of care reduces the po-tential benefits of care in the aggregite.,he an-swer could also be in the affirmative frthelevel ofthe individualfpractitioni-client interaction if re-sponsibility for inefficiency falls, at least in part,on the practitioner. To justify this conclusion wecan draw on two arguments that I have alreadyused. First, inefficient use of resourcesasuggestslack of skill or judgment' in the conduct of care: itis a manifestation of "logical redimAncy. Skc-ondly, it is a Misallocation of resources, no less at ,

the individual level than in the aggregate. The (individual pays more than he ought to for care \-

qt

either immediately and directly, or in the future itsthe consequences of program inefficiency eventu-ally work their way back to burden the irliVItialwho must, ultimately, foot the bill. /

Not ever ne will agree that wasteful tart is cafeor poor qu y. It is nut Important to have agree-ment on th . What Is important is that, in evalua.dons of quality, the element or wastefulness hr

4'1/4 dearly identified and maimed. Once this is dense,$!e could be made either to keep it

rate or to merge it into an overall measure ofquality.

There_k_nti disagreement that the balance of.health benefits and risks is at the tore of qualivassessmeht. To do no harm is the most hallowed IAprecepts in the clinical tradition. No element ofcare should he used if its risks exceed its expectedbenefits. Similarly, the combination and seqtfanting of elements shotild realite the largest het titsrelative to the risks incurred. I shall return t6 thislast stipulation when I deal with the stcittegies ofcare. .

If quality is measured in trims of actual VPexpected benefits to health from a specified course

\of action, the relationship between costs and qual-ity depends on the manner in which the factors Idescribed above enter into care. 'Die eliminationor reduction of wastefulness will allow us to havethe same quality at lower cost, or higher quality atthe,-same cost. This also holds to the extent thatcurrent care includes components that carry un-

-..necessary risk. But the presence of such compo-nentsnents adds another, more sinister, aspect to thegs.icture. It is possible for care to actually deterior-

in quality as costs increase above a certainpo t, if the added services include a large enoughcomponent of high-risk, low-benefit items.

Let us assume that the elements of are areproduced and applied as efficiently as possible,and that only elements of care are used that have ademonstrable benefit. The relationship betweenquality-end costs, as more and more elements ofcare are added, is empirically undetermined-Onecan, however, reasonably assume that the mostbeneOcial elements are likely to be used first andthat, as care becomes increasingly elaboratethrough the addition of more and more elements,the relatiie gains in benefits and quality becomesmaller and sma n other words, there arediminishing ret ns and, in the end, very smalladvantage to be erived from further enrichmentsin care or addit nal quantities of it. If we'accePtthis picture. the question becomes: At whatlevel of inputs into care should the standard ofquality be set? If individual patients paid all thecosts and reaped all the benefits the problemwoultilke easily solved by saying that the. patientand his physician cap jointly decide at what pointalf added costs are not worth the added benefits. Ibilieve that patients and physicians are capable of

or

making such Judgments. using the relatively in.complete information available to them, anti whenthe ahrtnatives are rather clearly demarcated. Butin older to make is more wedge determinetion,the data on costs and benefits must he much, moreaccurate, and reducible to , a common U1111 ofcomparisonhtr 4141111de. dollars.

The solution we have described is clearly in themainstream of the (finical tradition beluse it rec-rognites that patients differ, not only in the . t illy"wait al" feAtures of their illness, but also i thevaluations ;hes place on the costs 01 care and itsexpected treneficii Carried it, an -e-xtrrr., , tthsImmolation would support the gut feeling mansc link Unix have that the quality of care rests tit. o,+many individual xariltions to permit a generalstandard. What prhcets it from going quite thatfat is the large area (4 agreement among patientsand prat titioners on the valuations placed on costsand benefits.

Under the solutiot proposed above:the qualityof tare at the collective level would be the sumtotal of the quality of care determined case by case.litl1 IhirTfIrtrUS1011 could be invalid if the costswere collectivized, for example through health in-surance ;'if the I tells of hcahh #arc were sharedby more than the i.ecipiero; and if there were some

mit

socially legitimate reason Tor valuing the health ofsome, for example children, more than the healthof othetis, for example the aged. To the extent thatthe collective interest differs from the individualinterest, and the health practitioner is made thecustodian of bola., serious strains are introducedinto the patient-prattitionet relationship, and thedefinition of quality postv moral dilemma.

Strategies of care

Patient t or is a planned activity which involvesthe choice of specific elements ol4are from a po-tentially large pool of such elements, and theirproper sequencing in order to achieve specifieddiagnostic and iherapeuslit objectistes. A plan of lc-lion, as well as the panels-it of actions it generates.may be referredito as a strategy. A very simpleexample can be ssenIbled from the work of sev-eral investigators who Ave dealt with the choice ofstrategies for the management of acute pharyngitisin childrep and adults. '

According to Brook, the development of processcriteria in cooperation with infectious disease ex-perts resulted in the following - recommendationsfor the treltment of adults with sore throats:

"If history and physical indicate eukclence of a sorethroat, a throat culture should be clotie. If the cyltureis positive, she patient should make a repeq viawithin 24 hours and be given a shot of procaine(short-acting) penicillil; he should then wait 30 min-utes in the office so thlt he caribe promptly treated ifan anaphylactic reactibn occurs. After another 24

-1-1 al. 1

hours, the pandit should 'Immo. It no print Olin

Allergy has 'limn kora null 111 the mire %cuing none,

he should he *tyros 01 shad beatiailinsi thing aciiugi

prisktIlin '

In the example filed hs rt Al a smiler

stresiegst is det Lard. milt the atithoins ,,t ryes.represtitt "optnital-spialitv of (Mr I Initsik in el

al. propose to frL whether-yr stnsltat silategs Is, infat I, optimal !iv corn pit! MR thi cc II v pot hent al

ittategies lot the panagrinrot of mote throat tit

prisons who eh) i in( Iyavr 11111)1 s I11 1,(11 .111111) 41

trigs level I he dove abr. DallVes

4 re

-A ihr a anielith leoullnatral :11.urgi. an intim,.the lin nut la All 11.4(trnt ...41 fit 11/".4/ 141111 V1111111

111 ilifille Volilli1C a 'Muir. Mg' 11411111/41' 11, gt (mil A

tutrptut IH I I.II he rat all imi irony wai prim illm %.. uhtun tiwg a

1111)141 t ninny..t nrilltrt II ulitite nut 11c.11 .1111 nalic111 I

1.0 1 otripli4 air matteis !indict, these so mews atetested foi epicleinn versus rildenin (Ha all 1(11(1' of

the clisrase, lot oial set sus paienteial porn t illin,

and for chi leurrit espy( teal tiles all ihittive I hi nal(-tailor in the population a rpm (nig lot a ;ate.

1;111A1 II) has tamale the Inn ificatinli of itategies\Itaut the management ol at lite ph.1 t s ng tv rsru

more complex by itittodin Mg as a hist- step rules

that. on the basis of (Italica, rindings. (Lissa,. a airs

as ( I ) most probably (auscal In group A betahemolytic streplocmc its. (2) most pi obabls not so

(aimed. and (3).(plestionable as to ethology', With ;I

further subdivision Of (awl into adults and c hil-

alien six categeiries of patients ate 1111'1101rd: three

by etiologs and two In age. For rat Is of these it ispossible (I) to treat all cases without prior ( tailor-ing. (2) to culture and then neat, and (3) not totreat at all. As to treatment with print illin, it can

be either anal or parentetal Illus. their is a

matrix of 36 possible combinations, and for Cal 11 is

is possible to test the consequemes.iElaborate as )I f t mhis sees to be,thct\experienced)

clinician will Fri qoile, it as an oversimpliticatitutof a situation that is itself rlatisels simple. Morecomplete specification of strategies requires theconstruction of rather elaborate protocols, al-

' gorithms or trees. Some reference to this

work will be made in a subsequent section. All I

want to do do now is to la v. down the foundationfor asserting that the description and assessmentof the elements of care, one at a time, misses the

design, the rationale and the implications of thestrategy as a yehtde. including the consequences oftaking as well as pot taking certain actions. In myopinion, the very essence of quality, that elusivebut all-important ingredient that we (-all clinic-aljudgment., resides in the choice of the most appro-

priate strategy for the' management of any given

situation. I also bell ve that we now have thenecessary tools for tipecifying and testing such

12

i .

sit alrgiroi, whit 11 memos 11141 the mysteries of dine14) lodgment ,rte amenable to ieliling their clerkest se, 1 els I hrsr owls Air Oct ision A11444111 Mid

the analysis 411 (nil el let tiveness Alld isatthellerhe.

lint, although these tools at AVilildhle, the ilatathat 41 r needed tin then poet-tie application aregrowl ails 141 king, We need more Alt mate inlOrf1114114 .41ut I he' as t "mem r of illness and ofa 11nt al and lahni 41.11 v hinting% in association with

sot It illitov; ahittot the Mollehil y 1 MIS and othermks assay laird with diagnostit ',mediae*. orVal 4011s sequences of them, in t ortectiv identifyingilliir when it elitist% anti el I ntiellUsly missingrusting illness. ,ifAnit the ria,,,s, frordent- and-tionirt.o S Imo, ad .thrIluilve therapies as rt)Mpared to 011'11 1 1)1111 a/11111)11S In health; Acne the'dative valuations to br placed on various man-ifestations ad health or ill health so that these talibe added up ism; a weighted sum that act taillely

telly. Is then total impact; and, where costbeuefit r1 111111411111111 .ter to be made, a MI111111111 unit of /IllrallIlleliteln is Ileedet that permits a comparison/Even when all the In ii inatnut needed is not

lh

available. 111411V klIISIg11111 11111) 1111111M judgment canbe gained by making use of what is already knawn,supplemented by the opinions and valuations ofthose tv. II note prat time is imalt.r study, and of theirpatients.

Ot1V101114. a (lest ription of the techniques which

we have mentioned is beyond the scope 7of thispaper. Lusted has provided a reasonably simple

exposition of the clinical applications of decision II

analysis.' A recent paper by McNeil et al. may

with an aptittu e for quantitative methods istserve as a quick intro( iction.a The more ambitious

readerreferred to a. more rigorous exposition by Raiffa.'kiarinato has published a brief account of cost-effectiveness anti cost-benefit analysis as applied to

medical technology, anal provided an excellentbibliography.' Further discussion of the basic .

methods referred 10 a nd a large number ofapplations to 111C ass s ent of strategies irr----

surgical care, . to he In in an excellent bookedited by Bun er et al.' I refer to other workin a subsequent secti of this paper; but infly-The

surface tip be s med. iince this is an areacnrrently tinder intensive investigation, We are, at

last, experiencing a major advance in our under-standing of key elements in the quality of care. Itis a very phlegmatic person indeed who will not be

stirred as he first sees, as it were frcim a moun-taintop, this new and enchanting landscape!

Structure, process, outcome

In our framC of reference, when a judgment ismade on the quality of care it is taken to be, bydefinition, a judgment primarily on what profes-sionals do, and how they behave, as they interact

-directly with their patients.,'Hence, it is the processof sate hat is: ultimately; the object of quality as-,iessme t. Quality is defined as the degrecof con-formance to, or deviaiiOn from, normative. be-.havior. 'this formulation, both structural attrib-utes arfd outcomes areindirect means cif obraininginformation About the hormativeness of prifiCess.',The rather secondary role assigned to the as-

sessment of structure. is not to- be 'seriously.ettallertged.'ACcordirigly,In4rder to ynal e my task'Manageable,:l shall have little to say. about struc-turein, this paper; althOugh it will;not be entirely-precluded. Tho,situation is eidicrely different 'with

.resnestao__outsomos,, since,-_ according to ,a --large-body of opinion, including that 'oft inany leading, experts lualify.asternent, it Lsthe outcome Ofcare that is'the pri miry objectt.concern, andprocess 'only' a means_ to -the attainmentcome. It is ,with'clifficlenc.e, 'and with some-apology,

'that a Methodbased on process. cap,I/entitle inta:this hostile enyiron'tnenwrhereas_it is -'a proud badge OC,hondr, assuring- alnadst instant'attention and respect, to 'say, dfat a inethodt."Outcome,oriented$3,.;Some have ito hesitation toeven distort reality, relabeling process elements asotftcomes, in orcleX to avoid . the 'obloquy thatattaches to pioc-ess Aid to bay in the approbatiunthit;outdomes confer.. Of course, this picture islinveidr,aw,n, but not by much!

It is not true that outcomes are a more valid;mtasure-of,quality than is process, as it. is faihion-

sable to say. It is true that proceSS -pleasures arevalid indicators of quality only to the extent that,they relate to relevant outcomes. But it is equally

"- true that outcomes are valid measures of qualityonly to the extent that they relate to the antece-dent process of care. Fundamentally, 'validity de-,pends on the strength of the relationship betweenprocess and outcome, and on our understandingof that regionship. If that vital link is weak; or indoubt, Reither process nor outcome can be used toastessTqu-ality; the validity of both is attenuated,and to an equal -degree. This, means that, in 'this

: instance, we do not know what constitutes qulity.Further'. research is needed; the link befweenprocesi and:outcome must be stigated. In suchan investigation outcomes are the,,only reasonablecriterion. There can be-n<disagreement aboutthat, provided the proper outcomes are selected'and appropriately measured.

,11

ing knowledge of that relationsheis used to olkdin information about the, behavior of profes-sional personnel or of the larger sYstem- I

ABut, dogs monitoring have a subsidiary risearcir ,

function? Is "'monitoring analbgous to clini(il_medicine, where a physician Learns how to managecases by observing the outcomes of cafe? I thinkthe answer is bOth 'rtes" and ':No." It is "Yes", itra.restricted sense: if 'the. occurrence of unekpectedgood, or bad ottcomes leads*to '4 review of pfoCess" ,'in the light' or aken fly .krlwn redlio.nshipi -I,:,e-tween the two; and if, iik the event cutrent sknowl-edge does- not iirov,ide.an answer, questions' are

. raised about that knowledge so at,t,o suggest fur-, tiler research. per answer is alinott always "No," if.1

we we 'expect tile -moniteri eclianism_itself to'.bethat, further research :' Th establishment of new,,

---.--iinkages 6etween process OM outcome-can only be-achieved,.With'illy .gertatuty,, through carefully --controlled and meticulously 'conducted, clirticaltrials ..,It is unreasonalste and ,iclingtiions So. expecttterery PSRO.

hat tells us w good medicineor its analogu can asa

research'agency'tis. One might wiihetual reason a Frt, as has beendone to the `past, that: the best teskofthe useful-ness ora drug is the sum of the.judgmehts' of indi- .. t;-vidual physician's as they y-okderve its effects on the

- management of 'their individual patients. .0.,./4.1

Monitoring versus researeh

It seems to me..that much of the emphasis on theprimacy of outcomes in quality assessment arisesfrom a fundamental confusion between researchand monitoring. In research, new knowledge issought about the relationship between outcomeand process. In assessment and monitoring, exist-.

'The uses of outcomes

Nothing I have said sO'far Should be taken tomean that,outcomes a4 not important iniqualityassessment and monitoring.,puite the reverse istrue: Lei me count the ways.

Outcome-s, usually undesirable ones, can be usedas 'a method, of sampling 'or screening in orderto increase the yield of process assessment byconcentrating on cases with such ,,butcornes.Perhaps the clearest example of thiS strategy itto b found in the "problem-status outcomemethod described by MuShlin et al.9 It is also vis-

i_ible as an important element in the Performance:Evaluation Procedure (PEP) advocated by the

4 Joint Commission on Accreditation of liospi-tals.10

2. Outcomes can be tied as a proxy for elementsof process which are difficult: to treasure, orabout which information is hard to get or is ab-sent, provided the causal linkage between out-

, come and process is reasonably well established.3. Outcome items can also serve as a supplement to

monitoring process in order to ensure that im-portant process elements have not been over-lodked. In this case, they provide an added tier,so to speak, in the s.eillance system.

4. Outcomes can also serve as a feedback.amechanism that may lead to questions not only

about whether certain process 'elements are

r \

8

c

"adequately represented in,Toricoring but, morefundamentally, about assumed relationships bytweet] proceis:and outcome. As previously dis-cussed; this latter is the point where-monitoringand/research are most likely to intersect.

5 The inclusion of outcome assessment can alsoserve to reinforce problem=orierited' ,manage-

.. ment, or "management by objectii.res."11

6. Attention to the' attainment of prestated out-conies can motivate a more serious exatniation'of process and be a powerful spur Iii reform. In

... . Williamson's work on "health accounting," forexample, one finds not only an-implicit prefer-

.ence for management by objectives, but, ''also, a

. deliberate reliance on confrontation with fa ire, irrordar to jolt physidans, into action.' r7-Outcomes reflect the impact of all the elements'Na.! go intecare,_ and of much else besides.'They have, therefore, an integratiwe, propertywhich allows them to represent the contribution

.- 'of all the healtS prOfessionals to patient care.They also triclude the contribution of the patient.to his own care. Unfortunately, this useful ay-ity to pull together all these influences is also aweakness, if cknewishes to explain how the ob-served outcomes come about.'

8. Outcomes can not only include the patient's con-tribution to care, but also provide a means to..en-list the client in the process of quality assessmentand monitoring. The specification of the techni-cal elements of, process is an esoteric profes-sional enterprise about which the client can onlyhave rudimentary and, possibly, distotttedknowledge. By contrast, the patient ,has a greatdeal to say about the interpersonal componentin the process of care. He has at least as much tosay about the outcomes he expects, to what ex-tent these are attained, and how different out-comes are valued relative tos.each other and to

re costs incurred' in attaining them.I have been successful in my exploration of

the role of process and outcome in quality assess-ment, one should never again hear a preferencefor one, over the 'other, except by the poorly in-formell. But, I am also realistic enough to knowhow forlorn a hope this is. No doubt the debatewill go on. In the meantime, it:` may be useful topiesent some general principles that govern theuse of outcomes in quality assessment.

1. Obviously, the outcomes selected should berelevant to the goals of health care in any par-ticular situation.

2. This al,o means that the outcomes must beachievable by giiod care and that, the in-strumentalities needed are available to, andunder the control of, those whose performanceis being assessed.

S. The outcomes must be. specifiable in ,mag-nitude, frequency and timing. The paucity ofinformation about the course of untreated and

6 .

I

,

treated disease may makg it iinpossible tospecify accurately future .outcomes, limitingtheir-use in certainassessment schemes:."The duratio9 of outcomes as well as theirmagnitude must be taken into account. Hence

asemphasison a long-term perspective, 'often

as long as a lifetime, ind, the measurement ofhealth stattis.' 4

5. As 'a corollary to the aboie, pne needs toconsider the possibility -of trade-affs betweenmagnitude and duration for any one outcome,-

)and among severs) outcomes. For example ashoertr,,life.at a higher level of function mayhave: to .be weighed against. a ,longer life -bur-dened with greater disability.

t. As ano Pier corollary, accurate information)on'.

\ the ant ottlAtikes must be available andthe ou omits Must be subject to 'reasonablyprecise measurement. Since. definitive:out-comes- rhay 'not appear except after longperiods of time, inforMation may be hard toget:

7. It is necessary to examine the consequences notonly of taking-a specified action but, also, ofnot

m

taking such actiot 'in order to get acomplete picture. Thus, in evaluating the ef-fectiveness of a sargicarprocedure; its is neces-sary to follow not only those who have had,operatiois but also those whO might have beencandidates but were not. . 8

.

8. The attainment of outcomes cannot standalone as a measure of success. The means usedin achieving. these outcomes have; also to beconsidered, unless it is assumed that resourcesare un'imited, which is far from, true. Thissimple truth hasj-only occasionally been ap,7_;predated in assessments of quality, as distinctfrom utilization review;35 but it has receivedmuch attention in program evaluation, under,the heading of cost-effectiveness and cost-benefit analysis.' In this context, it is useful toremember that the costs are not only in re-sources expended or risks taken; any assaulton established values and social norms is also a'cost which deserves serious attention, unless itisr regarded as desirable and considered to' be a%

benefit.9. At an equally fundamental level, the problem

of attrib has-to be handled. Outcomes,whether od or bad, must be attributable,first, to medical care and, then,' to the per-forniance of those unde- r .,assessment. Natu-rally, the longer the time that has elapsed be-tween a specifipd activity and its consequence,the more opportunity there has been 'for theintervention of other factors, and the more dif-ficult it becomes to assign responsibility for ob-served outcomes. /

10. To conclude this, decalogue, when adverse out-comes are used to assess quality, one must al-

14

most always, examine the antecedent process/ofcare to find out what Went wrong, and hOw itmight be corrected. The search for causes and 'remedies, will often lead, ever) beyond process,to an examination of tire.; ural characteris-tics that have encouraged *scouraged speci-fied behaviors.,The quality:of care cannot befully comprehkuded Or successfullyared

a without understanding how- structure infl*ences protess,and process influences outcome,No*atter where one starts in this chain!, onemust ultiniately deal with it as a w,}Kile.

f

,

15

f.

O

7

Proposals for research

8 l begar ihis paper with a' general frame of ref-.

ere,nce hoping -that it would give meaning andperspective to the listing 'of research proposals;but, also, that it would generate a corresponding.classification. Unfortunately, no truly satisfactoryclassification was found; because there is a greatdpal.of overlap among the several categories thatemerged. Nevertheless, I hope that the followingis a reasonably orderly presentation in which thereader can discern the major features described inthe introducston. There is also some attempt tohave a very rough progression from studies deal-ing with basic concepts and measurement tools, tothose that deal with implementation; but too muchshould not be .made of this. Finally, there Is'theproblem of specification. Obviously, it is not possi-ble to be very specific in a general review like this.

The text only sketches areas for research. But,whenever I could, I have cited examples of re-ported studies that might serve as more detailedmodels. In these the reader will find not only con-crete embodiments of the more general descrip-tions in the text, but also; I hope, the raw materialand inspiration that generates new research. Sincethe studies cited are meant only to be illustrative,the list of references does not serve as a systematic

bibliography..

Basic explorations and studies of what consti-tutes quality

Specification and assessment by modeling As afirst step in exploring what constitutes quality, at

ast in the technical sense, diagnostic anderapeutic management should be described and

assessed as planned and sequenced activities orstrategies. The models of such strategies thatemerge will have a certain newness, although they k

use only existing knowledge and opinion. This in-formation is obtain d from reviews of the litera-ture, from retrospective review of case records,from the opinions and values of expert clinicians,.,

f and from reasonable extrapolations from all of theabove. The models that are construct& Pti the

Morin of algorithms or decision trees, are not onlymorp precise and realis(ic representatioris of whatis c nsidered to be goods 9r , but also provide ihe .,

' "opportunity to test altern ve strategies and toconfirm or modify normatie standards.. Suchformal testing is/necessary b use in these corn-plex'situations, the best course of action that intui-tion dictates may not be the best indfcated by deci-

sion analysis. ,

i.... The usefulness of this effort, then', is-that it pro-vides the basis for formulating criteria for assess-ing the quality of care. The models are also an im-

por.tant tool in medical education, as' a vehicle forspecifying and communicating to studen the in-tellectual operations that constitute cli I judg-

ment. Moreover, th -attempt to construct suchmodels reveals deft 'encies in current information,identifies th st critical defects, and suggeststhe research 'needed. to obtain the requiredinformation.

Illustrative, examples of this apprOach to the'.exploration of diagnostic strategies may be foundin the work of McNeil et al. on' screening fqrhypertension" and on diagnosing pulmonary e&;bolism;22 and in the work of Neutra on thedecision to operate for apRendicitis.." Examples ofthe specifitation and testing of more completestrategies that include -treatment as well as diag-nosis are pr ided by Schwartz et ak for essential

hypertep d renal artery disease;" by McNeil

Adelste for hypertensive renovasculardisease;2° and by Tompkins et al. for acutepharyngifis.2

A review of these publications illustrates the im-portance of costs as a factor i4 the analysis. Non-monetary 'cost' in the form of risk is always a factor.In many instances monetary cost also figuresprominently, for example in the papers by McNeil

et al." " and Tompkins et al.2 The problem ofplacing values on' the consequences of variouscourses of action bedevils the analysis. by Schwartz

et al)" Methods for valuing the quality of life,.in .addition to its mere duration, are briefly discussed

"by Abt.....- ,

r `

Even a nodding acquaintance with this work.. suggests opportunities for further research, be-

16

nlhtl its extension to cover a wider range of`51nical situations. wOne line of inquiry is to go

,Yond determining how to best arrive at a certainmanner in which cases .lugnosii by studying

resent clinically ide tillable problems can be---3'itned several diagnoses. The work of Ginsbergmay serve as an illustration.22 Another interestingstudy

would.be compa'ring tire stepwise, logicallyDaapproach to diagnosis by the "shot-

gun' approach; which uses a very much largerfitnnher of tests, at least as a first step, to quicklyb 4rowdewe nthe realistic alternatives. The possi-

gains in time and in the discdvery of illnessshouldbe weighed against the monetary and non-

m o netary costs.I

.. E 3

14Pirieal testing of strategies oreaire Modeling,wrie'ts Illatter how ingenious or creative, is limited by

' -.0'44' iS curr4ntlY known or believed about the rela-tionshipprocess and outcome. Additional

tli:cellviedge must come from empirical studies ofelements of care or the more cdmplexstrategies of management. This may b ccom-

:r Srled by observing what might be called 'naturalexpriments,.; but definiEive conclusions can onlyme from controlled trials, which cap b9 re-

g Ned as an extension of research in clinicalmedicine. gut clinical research is primarily cdn- l'

Rcirhed with .the comparative effectiveness of alter-natives; relative costliness, or the balance of costsandbenefits, is seldom an issue. When clinicaltrialsexpand their scope to include not only risks

''-`tlincl benefits td health, but also monetary costs andtiellerits, they 'address questions of social policy

at are also essential ingredients in quality.asset-11-8thent.-1.

"e algorithms that currently define acceptablePractice derive their authority from expert °Erin-ion, The work of Meyers et al. is one rather

41.1a1 example that is a 'reasonably direct ,em-1,31:rical test of such an algorithm: one fof thediagnosis of meningitis in children.23 Some-of thework cited earlier, for example that of MCNeil etal. tel,

4 the diagnosis' of pulmonary embolism, can'be regarded in the same light, since it,uses patientdiag-nostic though retroactively, to construct a diag-

Stu ri kt ra teg Y . 1 7 A modest proipective study by,), t-evant and Stern that tests the ability ofr-Ysiciarts to predict the finding of a stone oncholec

YstograPhY .is relevant because- analogousesti ,tes are used in decision arialysis, when nodata ate available.24 An example of a muchmore mbitious clinical trial that tests conse-

foquettc_.8 .utig; additional to, medical outcomes .may be

in tha,livork of Piachaud and Weddell on the

"lecono,

ics. aPtreating 'varicose veind.25 In thisstudy

scleroti..tbe,relaAi

tive merits of injection-cortipressionerapy and surgery were compared incontrolled

Ned eXPe

reCortis,

bette

riment, using as criteria the condi=

tio-n of the I b at the end of three years, mortalityfrom the procedure, the occurrence of immediatecomplications, the loss of patients due to non-attendance, the current monetary costs of theprocedures, and t it indirect cogi as representedby travel time for t atments, dayptaken off workre..and loss of earni s. There was no attempt toconstruct 1 singl measure for costs,and anotherfor benefits, but the advantage was clearly withsclerotherapy. The implications of such studies touality assessment and to social policy are obvious

And important, Lssuming that their findings can beaccept as valid.

IncremInts of cost, and of quality As I tried toshow in"the introductory section of this paper, therelationship between irwzements of,quality and ofcost is a matter that deseNes special attention. Oneway of studying it is through modeling and simula-tion. Using this method, McNeil et al. have esti-mated that it costs $8,485 to identify the last oneout of 20 cases of pulmonary embolism in a'youngadult population with pleuritic pain, whereas thepreceding 19 cases could be identified at an aver-age cost of $595 per case.'7 Even more striking prethe cost stimates of Neuhauser and Lewicki fdr aproce te for finding cases of cancer of the colon

hypothetiCal population of 10,000 persons 40years old or older." The procedure is to first do a'stool gimiac test and afterwards to x -ray, tho whoshow at least one positive test. What is varied henumber of successive guaiac tests that are incl edin the first step. Obvi?iusly, as the number of uchtests is increased, the number ofcases who sho atleast one positive test increases; more cases witcancer are detecte4, but there are also many morefalse alarms to be allayed by subsequent bariumenema. Accordingly, the cost of finding' an addi-tional case of cancer rises steeply as the "quality"of care, judged by the amber of stool guaiacsprescribed, increases; so much,so that the cost offinding the additional Case is_ estimated at an as-tounding $47 million when 6 tests are done,whereas it is only about $1,000 when one test isused as the Screen, and about $49,000 when three'tests are used. The values that are generated bythese models are, of course, heavily influenced bytb models' assumptions; and they could be inac-

'curate. My intent is to emphasize not the findings,but the nature of the method and its utility.

As an extension of such explorations, under es-sentially hypothetical conditions, itynight be usefulto have clinicians formUlate a set of ,,increasinglystringent criteria of the quality of management, aswell as an estimate of the gains in health expectedfrom such stepwise increments of quality. Cost es-timates could then be made, and the expectedbenefit compared to 'the cost. Hardwick et al. im-

vieTnented what is in effect, the firSt phase of such

9

a procedure'when they asked house staTeand gen-eral practitioners to specify the diagnositc teststhey would perform for specified conditions (1) a,,1

an absolute minimum, (2) routinely, (3) if .tie casewere&to be presented at a medical gran lc:kinds,

and (4) if it were to be included researchstudy.27

Needless to say, embirical studies are necessaryto establish the relationships between quality andcost. A study of secular, trends in the inputs intocare for carefully specified conditions as comparedto the outcomes of such care could be one initialapproack-A. good example is the study by 'Martinet al. of inputs into coronary care compared to the

10mortality during hospital stay.28 While such studiesare of great interest, contemporaneous compari-sons of costs for similar services at different levelsof quality may be' more credible. In one suchstudy, Jackson and Smith found no relationshipbetween estimates of the quality of pharmaceuticalservices in community pharmacies and the chargeper pre ripti .22 Similarly, Schroeder et al.fot*sd no, r relationship between the cost oflaborator7 services ordered by residents and theassessment of the competence of those residents bytheir supervisors, even though thereJwere widevariations in both estimate§." Rubenstein et al.have reported' findings that suggest diminishingreturns in outcome thtingswiih increments in thequality, of the process{ of:care.31 These are only ahandful of studies, but they show the way formore.

'Studies of the client -practitioner relationshipThe research I ...have _proposed so far definesquality in terms of the wchnical elenients in clini-cal Judgment. There is a corresponding universeof concern with the judicious management of theinterpersonal relationship. between clients andRractition.ers. Here, 'also, there are differingmodes of interaction and varying styles andstrategies of management that are more or lesssuccessful in achieving deOrable outcomes. Theseoutcomes include satisfaction, knowledge, changein attitudes and behavior, including compliancewith recommendations. All these mays, in time, berelated to success or failure in altering healthstatus. Fundamentally, there is little differencebetween studying -the effectS of two drugs on'hypertension and a study of the effects of two waysof managing the interpersonal. relatiorfship oncompliance with the drug regimen itself. Both areclinical trials, though they may draw on different

e bodies of theory and concepts. But, despite the.similarity, the study of' the effect of drugs isconsidered, squarely within the domain of clinicalpractice, whereas no one is quite sure where thestudy of the interpersonal relationship belongs.Ideally, clinicians should be as interested in one as

in the other, and actively engaged in research iboth.

The literature pertaining to the clientPractitioner relationship is too vast to permit aquick summary..A text book review by Bloom andWilson serves as a useful introduction." Lebowhas reviewed the literature on patient satisfaction;discussed some of the problems of method, andmade suggestions for further research." A specificexample of a clinical trial that examines the- conse-quences of a change in the client-practitioner 'in-teraction is the study by Inui et al.,- which shows

considerable improvement_in the-control of hyper--tension wher4hysicians shift their attention froin:the manifestations of the disease to thGtbehavior ofthe patients." These encouraging findings can beplaced against the discouraging experleztce inanother, cAnical trial of the lack of success witheither of two strategies, "augmented convenience"Or "mastery learning," to improve medicationcompliance in hypertension."

Views of what constitutes quality The research wehave reviewed and pcoposed so far is expected tpgive a mordprecisP answer to' the question: Whatconstitutes quality?, Another approach' is to studythe opinions held,,on this subject by clients,, ad-minisa-ators and professionals. A* comparison'should be made of the attributes that are believedto constituitnuality and of their rankings inimportance. The findings would -be germane notonly to quality assessment, but also to under-standing problems in system performance.

An example that fits precisely in this category-isthe study by Smith and Metzner of the opinions ofpatients, physicians and nurses on what constitutesquality of care in prepaid group practice." Mar--ram's work on how' much credence nurses are will-ing to give to the opinions of patients concerningdifferent aspects of nursing performance suggestsan interesting topic for further study." The litera-ture on patient satisfaction to which we have al-

ready referred includes much material from whichinferencei may be drawn about how,clients viewthe quality ofcare. In this respect, patient satisfac-tion has an interesting dual nature: it can be re-garded not only as an outcome of care, but also asa judgment by the client on the quality of care.

Continuity and coordination, as attributes ofquality There is a general presumption that thecontinuity and coordinationrof care are impostentconsiderations in the assesiment of quality, eventhough it is not quite clear what these phenomenaare, or how they fit into a formulation of whatquality is. De. Geyndt accords them a central placein the definition of quality by considering them to

1.8

be aspects of the4rocess" of e as distinct fromits "content."" I have preferr re them asan aspect of the organization of whar I h4ve called'`,process. "" More portant than these formaldistinctio,vs is the pr cise definition and measure-ment of these, attri utes, and the study of theircontribution to th Outcomes of care. EmpiricalTrudy should also include the degree to whichexisting differences in, Or purp ive manipulationsof, certain structural features mg' about differ-ences in continuity and coordina ion of care. I

would also like to see/ a test of the hypothesis thatplanned tjAnsfers of the responsibility o£ carefrom one physician to another might actuallyimprove care by creating an opportunity to assessit anew, and by making the performance of physi-cians more visible among colleagues.

Important Contributions to a more peecise defi-nition of the concept of continuity may be foundin papers by Shorten" and by Bass and Wi'ndle."However, the tendency to.4eraitiOnalize con-tinuity to mean care by one fhyNician or. a singlesource of care, except by referral, does not seem tome to capture the essence of the concept ofcontinuity. Almost all the studies I have seen haveused this definition."42-47 Bass and indle tried toassess the relatedness of past to present care."Shorr and Nutting defined continuity as the com-pletion of a needed sequence of are." Becker etal. describe a well-controlled tri of the effects Of--7having children see the same physician at eachclinic visit, and give a gobd review of the relevantliterature."

Description of prevalent patternsof care

and strategies

In a Previous section we discussed th modelingand testing of strategies of care. In this section wefocus on a complement to the research describedearlier, namely on the identification of how physi-cians behave in the real world, described in termsof the elements of care, as well as bundles andconfigurations of such elements. Here we seek todescribe what goes on, what factors. influence it,and what the consequences seem to be.

Studies of the elements of clinical behas;.ior Muchmoil information is needed about differencrs inthe patterns of care among practitioners within a.given setting and across settings. We need to,understand what, factors are reorisible for suchdifferences and what the costs and other corise-quences 'are: The factors invplved could inckidepersonality attributes, knowledge, training andsocialization, position within an organization, re-sponse to role models within the organization; andfinancial incentives.

Several studies of the use of laboratory and .

other diagnostic procedures an be cited as a goodexample of -this type of research. The findings ofChilds and Hunter. suggest that monetary returnOn lik.investment in an x-ray machine may be afactor in recommending radiological procedures.5°On the contrary, the °persistence of large differ-ences in the use of laboratory and, other services instwitg. s where there is no direct finanCiakincentiveto use such services," or where the incentive is thesame for all physicians, 52.53 suggests that Other fae-Cors are equally or more important. Schroeder andhis associates in a series of studies have looked intothe correlatObf .lifferences in, laboratory use andshown that these differences are net related to dif-ferences in corripctence," Or to differences in out-comes, or "proddetivity";" and thavphysiciansmay respond to a "cost audit" by red cing labora-tory use."

IdA ittfivition of , styles and strategies I Havealready defined a strategy as a plan 'for action, and

cussed the iIportance of dealing with strategiesin t definition and assessment of quality. A"style" lay be defined as a habitual preference forcertain ,modes of decision making which wouldmanifest itself in components of strategies orstrategies as a whole. For example, a physician mayexhibit-a persistent' and pervasive preference forerrors of commission over errors of-omission, pnemanifestation of which may be a large redundancyin gathering information-. He may give evidence ofmore than usually routinized or stereotyped be-

.havior. These and other persistent yet undefinedpropensities requiri precise conceptual formula-tion Ind empirical study. c.

The notions of style and strategy have applica-i-tion beyond the solution of clinical probleis. Ihave already suggested that they can also be usedto study the clie t-practitioner relationship. They

titionerhieve the

also apply to the way in which a'manages an enti case load, hoping to amost efficient allocation of his time, attention; andother resources among competing calls on them.

Styles and strategies can be inferred from physi-cian behavior in real-life situations or under moreartificial test con itions. Information on therationale employed y the physician can be ob-.tained more directly y having him explain, as heworks, thereasons f. doing what he does. Fattuhas summarized some f the early work using thismethod, known as rifle n /Attlee, as well as othermethods in studying pr lem solving." A morerecent example of the use of reflexion parlee inexploring clinical decision making has been re-ported by Kleinmuntz.57 A study of the diagnosticprocess by Leaper et a). notes variations in thedegree to which the1clinical interview is either"stereotyped" or "adapted" to the problems of

1Fach patient, as inferred by an observer."

Obviously, strategies That are identified throughempirical study become candidates for testing, asdescribed in the opening section on research pro-posals. I alsOolieve that strategies used by "good"pit ysicians are a 'more valid basis for the formula-tion of expliCit process criteria than is the practiceof having physicians list all the things that shouldbe donefor cases with a specified diagnosis.

,Comparisons- of norms with practice The litera-ture teems with observations that' physicians fallshort of the normative standards of care. In somecases it has, bee suggested that physicianS\do notfollow, the sta Ards which they thlemselves, as a

group, have rmulated." Some hale claimed thatmashy errors in care are due to inattention by theoverworked physician." Others have shown thatpart of the deficiency in performance is slue tolack of knowledge, while another part is due to notacting on what is known." In my opinion, theexplicit criteria lists which are often used to judgeperformance are, themselves, often faulty. Oneimportant inadequacy is that they fail to take 'ac-count of the many contingencies, including multi-ple diagnoses, that modify the strategies isf man-agement. Moreover, it may be inappropriate to-apply to office practice, especially to that of thegeneralist, criteria derived from strategies of man-agement that are suitable in 'academic settingswhere highly specialized physicians are involved inthe,polution of difficult diagnostic and therapeuticproblems. ,

Unfortunately, much of theabove is only conjec-

ture;ttire; d there is an urgent need for studies thatatte t to understand why physicians conducicare in the way they do, before passing judgmentthat what they do is inapppriate.

Homogeneity and heteroge sty of . performanceAn interesting question wit many practical impli-cations' is whether physicians and other practition-ers perform equally well across a range of activitiesand functions, or whether they do well in someand not so well in others. This involves examiningthe htogeneity of performance in the practice ofindivi al physicians, as well as the ability of aninstitution to 4-educe variation in practice acrossphysicians.

The relevance of this issue to quality assessmentis most apparent when it comes to sampling. Inone 'method, the "tracer methodology" proposedby Kessner, there is an explicit.assumption that theperformance of an entire system' can be mappedby judicious selection of a small number of condi-tions that can stand for all the rest." In manyother studies, when a all number of diseases orconditions are seleCted fo ssessment, there is asimilar assumption, even th ugh it may not be

2 0

explicit, or occupy such a central place in thedesign of the method. $ ,

Kessner has tested his assump/ion Ofhorpogeneity and found it, at bet, frail." LKonsand Payne have reviewed the literature and donefurther ,testing of the degree of compliance with`normative standaids of management by individualphysicians, across diagnoses, in office practice,"and in hospital-practice." s tendedto be low. There was, however, a sugg dot} thatgreater homogeneity might be found in the work"of phySician subgroups'who havea more restricteddomain. This clustering was also found in a Studyof the office care of a set of preventive and illness )

situations in children. Homogeneity of perform-ance was reasonably high within each of these :categories of conditions, and it wa,$ high r fcKpediatricians than for family physicians."

The epidemiology of quality

Variations in the quality of-care are, n t simply arandom phenomenon. They are highly terued,and responsive to causative factors ttfat we.need toidentify_ and understand if the quality of care is tobesuccessfully safeguarded. The first step in thisexploration is to.'answer the classic questions of

demiological investigation: How much? elVho?ere? and When? The results of these observa-

tions maff suggest answers to the most critical of allquestiobs: Why?,;The causal hypos eses thatemerge could, then, be tested through mor or-

ous observational studies and confirmed by actualexperimentation. But, for now, even\the simplestof deScriptive studies would add a great deal to. theNile we know.

On a larger scale, we can 74 say *lost nothing,.about the quality of care for the nation as a whole,or for reasonably large populations in thtr natu-ral 4abitats,otiher than what can be inferred from

;crtele mortality, morbidity and utilization data.The one exception to this generalization that Iknow about 'g the study of a segment of cars forthe residents of Hawaii by Payne and his as-sociates." We are similarly in the dark about timetrends: Is the quality of care improving, and howrapidly? Thii question is difficult to answer be-cautv it requires the separation of two phe-nomena: changes in the science and ,technology ofmedicine, and changes in the application of thatscience and technology. Both phenomena need tobe assessed.

The epidemiology of quality can be viewed asmanifesting itself in two populations: (1 the pro-yiders, and (2) the clients. 'Obviously, these are nottwo separate compartments. Variations in the qual-ity of care received by clients are probably largelydue to the kinds of providers who care for them. I

monde to what extent the reverse could also he the characteristics par-,of physicians that perform paril

true. , ticularly well or badly could be a useful way of. , generating hypotheses about the determinants of ,$

performance. f N

iAmong the recent studies that have attempted toA mecasure the magnitude of the effect of each of/ seieral variables on iperfnance are those of,

Rhee7' and of the Stanford University groupresponsible for the Institutionaltifferences Study.72 Rhee used data from the study by Payne et al."in' which the dependent variable was a perform-ance score based on compliance with explicit progb .-

ess criteria. Notable among his findings was thelarge effect of hospital,cliaracteristics compared to'the effcct of specializaition and of organization intolarge,nultispecialty groups; and, evepnore 13.striking, is the magnitude of the unexplainedvariance: In their study onpost-operative surgicalmortality and morbidity the Stanfol-d group., npt'only found unexplained variance of a similarorder of magnitude, but also failed to confirm the 'effect of factors usually thought to be conducive toquality, for example hospital size and universityaffiliation. Both of these studies can serve asmodels for future work. The Institutional Differ-ences Study is particularly notable for the methodsit developed to control for risk factors that influ-

'measure organizations variables. Its findingsence the outcome of s rgerr and td specify. and

could perhaps be better Aderstood it samples ofrecords in the hospitals involved were to be sub-jected to an assessment of the process cif care. Iwould accord such a study a high pri rity in thisprospectus.

' The correlations bitween organizati nal characteristics and perform&nce are, of course, extremglyimportant for system design. It would in interest-ing, therefore, to go.one step further and deter-

/mine whether the-same physicians will, alter theibehavior when placed in differeirit settings.-,

Tim- epidemiology of quang' among provider'The studies that might belong in this categoryoverlap.rith those I have already described in theptevious section.on Description of Prevalent Pat-.terns. If there'is a istinction at all, in the earliersection 1 .included tudies that dealt with thecletailed ontent of\ care and . the rationale that

.,, .esplaiteddifferences in that contentfiere, we are... more interested in who provides 'care of good, bad

or inclifferett quality, and under what circum-stances. Much of the literattire of quality assess-mept deals with this question; but the restrictedscope of most studies, and limitations in theirdesign and analytic methods, inakes generalizationhazardous. It is clear, however,' that performanceis related to attributes of practitioners, attributesdf the organiz*ional settings in which they work, .

Ind interactiOrA between the two. Among theattributes of the providers are education, training,specialization and length of practice. The role ofpersonality characteristics including motivation,while suspected to be large, has nocreai4ed muchattention, except in studies of the academic per-

' formancle of medical studenas. Among the attrib-utes of organiiatibnal settings, usually hospitals,that have been found'to be influential, perhaps the

. most important has been affiliation with a medical.ca school. Other attributes, are involvement in resi-

dency and internship training, auspices, sqffing,financing, size, and organizational contrIlls onstaff appointments -and activities. Unfortunately, -theie is far fr%n unanimity on whether thesefactors are influential and what their effects are.For example; we are still not certain whether theorganization of physicians into prepaid groups

Results in better quality, and, if so, to what extent.tThe 'numbe of studies that have attempted ac-Atually to qu ntify the influence of each factorseparately an in combination with others is par-tic larly small. Whenever this has been done, whatis ost impressive is the very small amount of

. v riance explained by the variables in the analysis,' suggesting that we still know very little about the

determinants of the quality of care.". The literature relevant to this section is so largethat even a partial review would be a herculeantask whiCh I shall not attempt. Only some .exam-ples will be given. Among the earlier studiesparticularly outstaistling is the work of Peterson etal." and of Morehead et al." Among the morerecent work is the Hawaii study by Payne and hisassociates," and the study of Medicaid benefici-aries in New Mexico by BrEioke and Williams." Of'particular inteiestin the latter is the special wen-.tion to "outlier" physicians, those whose perform-an.fic was markedly deviant.70 Further studies of

. 21

Epidemiology of quality' among clients Unl theemphasis on provider characteristics, there hasbeen very little attention to client characteristics instudies of the quality of care. Although the receiptof quality care by disadvantaged populations'is amatter of great social concern, and there is muchpublic debate about it, the information bear ng onthe question is indirect. It deals main withdifferences in levels' and patterns of utilization, insources of care, and in morbidity and mortalitydata. Without minimizing the relevance of such .information, it would be useful to have moredirect and definitive assessments of the quality ofcare received by persons differentiated by age, sex, .

educations, color, income, occupation and otherdemographic and socioeconomic variables. Muchof the differences would probably be related todifferences in sources of care. But a question that

---.

14

'

'needs to receive special attention is whether thesame institution, and the same practitioner, give'different types oriiare to patients with similac,

.0medtal conditions because of differences in thedemographic and socioeconomic characteristics ofthe latter. Some adaptations of care to suchcharactefistics are; of course, not onlylergitimatebut, also, desirable and necessary. The' issue,is todetermine *icttether the adaptations are made tomaintain a ,.high level of quality or-whether qualitysuffers.4 ,

The literature having' a bearing On uality ofcare for disadvantage populations-44,sbeen 're-viewed by Broo d !Hams." In another publi-cation they d scribe t eiown fiticlings in a studyof Medicaid eficiaries." Lyons and Payne h Yedescribed the re nship Izetween age and Iltequality of care in 'their studies ill Hawaii."'"

essner et al.- hasie described the ',relationship1K

socioeconomic factors, as well asbetween syrem performance and various demo-graphic at

of care, revealed 'by an application of the"tracer" method, in selected populations in theWashington, D.C. area.63 Griner and Liptzin havedescribed the effects of patient characteristics suchas age, insurance status and ward or privateaccommodation on the use of laboratory tests in ateaching ho s al." .

The ident cation of time trend's is an importanttool in epideviological analysis that has beenseldom employed in studies of quality. Hence, twostudies that have information on this subject areparticularly interesting. The first is a study ofsurvival after cancer of' the cervix uteri treated adecad apart. The findings suggest that improve-ments have been due mainly to a diffusion ofknaaedge from rrilijOr centers to community hos-

.

Ifftals, and to a much lesser extenCto an improve-ment in medical science." The second is a study ofmaternal mortality in Michigan from 1950 to 1971showing that, in spite of spectacular declines inmtirtality, the percent of deathsNonsidered "pre-

7:':-ventable" by the States maternal mortality corn-,mittee fias increased markedly, from about 60'percent to about 80-.percent." A retroactive reas-sessment of the Committee's file of cases, applyingcurrent standards of preventability could be veryrevealing when compared to the contemporaneous

' assessments.,,Most of the work on differences in the manage-

ment of patients by the same piovider, whether aninstitution or an individtial practitioner, has beendone in the field of psychiatric care. Perhaps thebest known of these studies is the work of Hol-lingshead and Redlich in New Haven," but thereare many others. In a later work, Duff and Hol-lingshead showed that such differences in care canalso be observed in a teaching hospital engaged. inproviding genepai medical care." The under-standable reluctance to look into this matter must

be o c e. No program that provides care toclients of widely varying background* can affordto igncir the possibility of discriminatory behaviorin the application 'of careNrrco*itravention of themost sacred traditions oLthe titling profestions. ".

#i relationship of structure to process orcome.' . t.

The eader will recognize that the epidemioloii:cal-stu es skeichefl out above often deal v#th.sob- 'served relationships beta Teen structural afarateristics (attributes of practitioners and instil -

dons) and process, or outcome, or both. The moredefinitive verification of such relationseips will re,-quire controlled experimentation. The major pur-pose of such studies is to safeguard and improvethe quality of care. However, at the same time,they can elucidate the operational meaning of cer-tain concepts, for example "continuity." Tothe ex-tent that attributes of structure are 'found to beregularly related to performance, the more gen-eral use 'of such attributes as measures of qualitywill be mot firmiblistied.

Development Of basic tools for assessment is

Many of the studies mentioned in previous sec-.tions, as well as many-still to bc_ described, cannotbe done well unless certain' bas c tools. a7 avail-able. Thus, the refinement of existing instruments,and the development and testing of new ones, is anecessary part of research in quality assessment.At issue are the reliability, validity and cost of thebasic instruments of assessment. A few of these,that I consider most important, *ill be selected forfurther attention.

Specification and measurement of outcomes. I

have already indicated the ways in which themeasurement of outcomes fits into quality assess-ment'. and monitoring. In this section, l shadescribe briefly some ways of measuring outco

One approach is to develop and use indicatorshealth and social well-being which permit a gen-eral oversight of a community of population. Thisapproath is typified by the "sentinel events" pro-posed by Rutstin et al." A useful area of researchwould be to specify appropriate indicators, de-velop methods for data collection, actually imple-ment data collection and interpretation, and de-'termine the usefulness of the information inbringing about change. Such a system would, ofcourse, have to be adapted to the special needs ofits users: whether, a planning body, a program, oran organization that provides care. Any of tifiese

agencies may need to supplement information

2-2 1`.

which it collc4s with 'information from otl-arsources, including census data and informationfrom.the.National Health Survey. Naturally, theindicistor conditions need not 1,4 only outcomes; avariety of process elensknircan, also be included.Moreover, the ,system will be relevant not only todetermination of quality, but also ay need andunmqt need, resource use, and so on. Many or-ginizations are,'Ipo doubt, already involved in datagathering activities of this general kind. Perhapsthe first step wouk be to review all that is beingdone, tg document Its current use, and to'assess itspotential usefulness._ It islimpbrt4nt to rixnemberthat a . system overladen' with data that are not

.useful or are not us4d, can be as bad as one thatgish4s t little information. In any event, a pause for

rea ment could be most helpful.A second line to pursue is the development of

"integrative" 'measures of health status that canrepresent" the outcome Of all the factors thatinfluence health. The distinctive features of such ameasure are that; (1)4he impact 9f mortality andmorbidity are combined; (2) morbidity is repre-sented by a gradation of mutually exclusivecategories of dysfun`Ction; (3) dysfunction includes'social and psychological, as well as physical, disabil-ity; and (4) the several dysfunctional states areweighted and summed into a single measure,-Theobject is to arrive at a summary representation ofthe quantity, and quality of life of a cohort of indi-viduals over a period of time, often a complete lifespan.

Elsewhere, I have briefly reviewed the earlierstages in the develypment of this approachthrough the work of Sanders,'Chiang,and Fanshell and Bush." Since then, this area ofendeavor hasoexperienced an almost explosivegrowth which includes further work by Bush andhis associates,"" the work of Torrance," and thework of Gilson and her associates on the SicknessImpact Profile." " Three collections of papers willprovide the reader with a concentrated and rea-sonably current overview of the field."

Perhaps the central problem in the constructionof integrative measures of health siatus is that ofvaluation. There is need for empirical studies ofthe valuations placed by clients on different de-grees and kinds of dysfunction. Expecially in-teresting would be differences in relative valuationby persons who are currently experiencing a par-ticular level of disability. The effect of length oftime in any level of dysfunction should also beexamined, as suggested by Torrance." Anotherline of development might be to try out .a totallydifferent method of valuation, comparison with astandard population, as proposed by Breslow andhis associates."

A third line of development is the constructionof integrative measures of health status that arecondition - specific. The global indices described

ii,above will plibably be found to be 4.cking in sen-

,sitiyity and specificity when used aI measures ofthe quality of ar . There is an opportunity toreihedy these ects by developing analogousmeasures of the uratio .. quality of life of per-sons suffering from specific c ditions, fbr,exam-ple a particular cancer. The mea ures of functionand dysfunction induct/et can th be tailored tothe condition, whivatten an gii n to includingmanifestations that can be pr ted or remediedby proper care. Thetesting of such measures willals provide an 'opportunity to study the course ok

ess and identify-Mdditional outcomerrat catserve as measures of quality.

A fourth line that might-be pursued under the 15general heading of outcome measures is the de-velopment of condition- specific of out-come. Here, only key indicators are used singly orin a profile; there is no attempt to integrate them,together with losses from mortality, into a singlemeasure, as envisaged in the preceding section.But, obviously, there is a relationship betiveen thetwo approaches, since the identification of indi-vidual indica rscmust be a step iff the constructionof an integrativ measure.

The time e sed since the institution care isan important classifying variable in o tcomestudies. Accordingly, the indicators of ou omecan be contemporaneous with care, br can f owcare, in which case they are proximate (short-term)or remote (long-term). Short-term outcomes areofspecial usefulness in monitoring becatir they canbe used to identify cases that require furtherstudy:. Thfris a method of venerable ancestry,going back to the _classic work of Codman," andearlier. Its more recent manifestations in,the workof Williamson and his students have already beennoted.'2 A particularly useful model of the kindof research and development needed for con-structing short -term indicators of outcome, is the 0

work recently reported by Brook et al." What stillremains as an essentially unsolved challenge is thedevelopment of concurrent monitoring, using out-comes during the process of care. Of course, theconduct of care from day to day is constantlyguided by such outcomes. The difficulty has beenin adapting this everyday occurrence to a formalsystem, of monitorrng, other than direct supervi-sion by peers or superiors. Some of the recent ad-vances i computer-aided management do; how-ever, su est a possitle solution." What seems, tobe necessary s the constant feeding of selected in-formation into a computerizedsystem which raisesan alarm when certain, prespecified, configura-tions of events occur or fail to occur during sped-,fled time intervals.

A final line of inquiry derives from adopting adefinition of quality that includes phenomena suchas satisfaction, attitudes, opinions, knowledge, ill-ness behavior, and the like. This opens up the23

16

whole a ea of methods in behavioral researchwhich ca be assessed'and implemented by inves-

tigators aving the necessary pleparation. Amodel for such research that is closer to home maybe foundA the work of Hulka (and herassociates,

first, in h development.of a scale of client satis-faction and,othen, in using that instrument' tostudy its epidemiology.'' -95

Improvement,' in the medical FKa6-1The medi-cal record is almost always fllie key documentwhich contains e information for assessment

of care. Judgn nts bf quality are heavily influ-enced by the nature of the record. There is alsothe possibility that the record is, itself, influencedby quality assessment activities.33.37 Unfortunately,in spite of its key role in patient care and its evalu-ation, the record is often inadequate or poorlyadapted:so these purposes, especially in ambula-tory care. The follow' ig are some proposals for

()remedying this situatiob. . ^;tWhile the record is often recognized to, be in-

complete, the accuracy of the nformation that itdoes contain is seldom questio ed. The early ,workof Lembcke" is an excepti t this generaliza-tion, as is the more recent w k of Wiener andNathanson." It would be usef to test, by seeking

. independent verification, th accuracy of the his-tory, physical findings, results of diagnostic tests,and so on. As a second step, it would be interestingto see what effect corrections of these errors wouldmake on an independent judgment of quality'based on a review of the record.

i The completeness, and some aspects of the accu-

racy, of the record can be studied by arranging forindependent direct observation of they client-practitioner encounter, or by recording it onvideotape. Use of the latter method has been re-ported by Turner et al.'°°, Zuckerman et al.,101 and

SteWard and Buck.'"Alternative ways of designing records should be

developed and tested as to their usefulness in themanagement of care as well as its assessment. Con-siderable work of this nature has been done inconnection with the problem-oriented record.Examples are the studies of Tufo et al.103 andSimborg et al.10' Other work has been reported byGrover and Greenberg.'°3 In work of this kind,

,.., the objectives include not only completeness andccuracy of information, but also ease- in finding

what is needed, and the ability to identify the prac-titioner's intent and reconstruct his rationale.Another objective might be the inclusion of thecontributions to care of nurses, social workers andother professionals, so that assessments can bemore inclusive. In a subsequent section I shall dealwith the feasibility of even including entries by the

patient himself..At a more fundamental level, we have generally

allwed e traditional content of the medical rec-ord to tate what is included or not included- in

assessme is of quality. In this way, the record con-strains the definition of quality, allowing the tail towag the dog. It seems to me that 1% is mote reasons

able to begin-by defining quality Independeptlyofthe record and, then, to design the record so that.alone or in combination withOther specifiedsources, it can provide the information that cprfe2sponds to the initial definition of quality. In suchan. enterprise there would be a great, temptation todemand an im/Sossibld degree of completeness.

, One must be adamant In resisting it. Th morereasonable and. challenging objective would be todefine and implement the near-minimal set thatwould permit p oper management and assessment.

I have alre y referred to the uses of com-puterizedputerized records in computer-aided management,and to the affinities between the latter and concur-rent monitoring of care. Setting out to design such

terns is an)excellent opportunity to rethink therecord and .adapt it more suitably to its severaluses. The releyant literature is immense, By way ofexamples, I have already referred to the work ofMcDonald" and of Schmidt et al." Other work in-cludes that of Wassertheil-Smoller et al.1" and .,Barnett et al.'" .

The criteria and standards of quality No assess-

ment.is possible without some.standard for com-parison. In studies of quality, two or more provid-ers may simply be compared. Another very ,preva-lent approach is to judge performance by theextent to which it attains normative standards.

The urgent need to develop realistic and validcriteria and standards for condition-specific out-comes is implied in my previous discussion of themeasurement of outcomes, and can be seen as aparallel activity. As Brook et al. have shown, thekey steps are: (1) the identificatinRof relevant out-comes; (2) ordering them by importance; r3..)-find-ing reliable and valid means to get informationabout the outcomes and to measure them; (4)specifying when dui.ing care or following it eachoutcome is to be measured, so that it is most dis-criminating (sensitive and specific) as a measure ofperformance; and (5) specifying the degree ofprogress toward each outcome that can be ex-pected by good care, given certain attribUtes of thepatient and his illness." In addition to using suchoutcomes for retrospective review of care, there isneed to develop methods for using 'them in con-current monitoring, as has already been discussed.

Criteria and standards for the assessment of theproCess of care are very widely prevalent and are abasic tool in current assessment, monitoring andcontrol activities. In spite of their central impor-tance as a measurement device, the criteria listshave attracted little serious scientific analysis: I

24

shall devote the, rest of this section to proposing .ways of remedying this deficiency.

perhaps the first' step is to develop a taxonomyof criteria lists and similar formulations based 9Iiikey attributes of their design and its underlyiiglogic. Next would come an analysis of the possibleimplications of these features for quality assess-dent. The twork of Rosenberg" is an example ofan initial exploration in this direction. Much morework is needed, and soon.

.

.,,,. As a result of the above, or independently, workshould proceed on developing and testing alterna-tive cr*ria designs. One way to go is to developalgorifc formats that define more precisely op-

, timal or acceptable strategies for management, tak-ing account of frequently encountered contiogen-cies. The worktof Greenfield and his associates isan excellent example." Initially, these algorithms

*derive their validity from etpert opinion. Ulti-.- 8 lively, they should lie tested emeirically, as indi-cated in an earlier section.

When a system of monitoring is designed, itmight be useful to consider the use of several setsof criteria in stepwise fashion. For example, a Sim-ple list could be used for screening, with a moreelaborate algorithm to be used for more definitive

s'udgments in cases that fail the screen, possiblyupplemented by a sample of cases that pass. Aombination of a concise algorithM with judgments

using "implicit" criteria may be tried out. Thework of Mnshlin provides an example of the lat-ter.' Rubenstein et al. give an example of how

. 'criteria with "laundry list" and algorithmic com-ponents can be combined into a "decision index. ""

As mentioned in the recedi g section there is a., mutual- interdependent betw en recording'and

assessment. Hence, o part of the effort to de-velop and use alter tive criteria designs is thewoirk needed to r esign the record so that thecriteria can be more efficiently applied.

The application of criteria lists to the assessmentof process results, initially, in a "profile" of indi-vidual criteria that are met or not met. The deriva-tion of an arithmetic average weights each itemequally. Differential weights may be assigned tothe several items and a weighted average obtained,as in the work of Payne et al." These procedureslead to several difficulties. First, a given score canbe obtained by different combinations of perforrh.ance and non - performance of the several criteriaon the list. Are these different combinationsequivalent, or are there combinatorial interactionsthat are missed, by simple summation? Second, wedon't know what any given numerical score means.Is a score of 65 "good," 'fair" or "ppor" ?.Third,we do not know for certain what the basis for theweighting is, and -how valid the weights are. Fi-ney, a related matter, the construction of an av-erage, weighted or unweighted, does not accordwith the intuitive view that in some instances theJ

Cri render?theabse ce of one critical element in care rende theentire care disastrous, no' matter how many brow-nie points the care can earn in other respects: Ofcourse, this could be handled by assigning near-infinite weight to such elements, provided the con-figurations that render them critical can be de-fined in advance.

a.".The problem of weighting could be investigatedthrough comparing the items on a criteria list withthe corresponding algorithm, and to both implicit'judgments of quality and `the outcomes of care.World that has'a bearing on the question of weight--ing includes that of Richardson," of Hoekelmanand Peters," of Lyons and Payne," of Hopkinset al.,"3 and of Novick et al"

Another interesting lire of inquiry would be tosubject a set of recoPd to as ssment Using differ-ent types of criteria. The co mplications of_ ac- ttually satisfying different type f criteria shouldalso be determined and compar d to the expectedand, where feasible, the actual outcomes'of care.

The social process, including group interaction,that leads to the formulation of, and agreement.on, explicit normative criteria has been, to myknowledge, an almost totally neglected area of re-search. It would be useful to know how leadershipis exercised, dissent handled and differences re-solved. The effect of including health profession-als other than physicians, and of administrators or,even, consumers should be studied. Similarly, theinclusion of physicians from a broder range ofspeOialties, for example psychiatry, physiatry orpublic health and preventive medicine, might havean interesting effect. The content of the criteria aswell as the process of arriving at them might beinfluenced. As a subset of such studies, it would beuseful toilook into 'ways of expediting the ocessof peer concensus and improving the dec sion bystaff work that provides necessary infor ation,forms or worksheets, and otherwise structures ofthe situation. Brook et al. demonstrate the Useful-ness of such staff work and, incidentally, commenton the impact of including a psychiatrist on thepanel dealing with the outcomes of breastsurgery.'3 We are indebted to these investigatorsfor giving us this information. There are otherworkrs in the field who also have considerableexperience in such matters, but who have nottaken the time to describe it for publication,perhaps because they have not realized how im-portant it is. 'As a simple first step, I would suggestthat this Bind of information be tapped, even if itproduces only descriptive. accounts .end informedguesses about what works and what does not

2

/I

17

Monetary measures of costs and benefits Theneed to measure the monetary costs of inputs andthe monetary equivalent of ben fits arises fre-quently in assessing quality, as we have seen

25

18.

already. Precise and valid cost and benefit meas-urements are also required to assess the

monitoringof utilization control and qualitysystems. In addition to devising rigorous andstandardized cost accounting procedures, thereare some important conceptual problems that re-quire attention. The problem of arriving at mone-tary equivalents for nonmonetary costs and bene-

fits has alreadyleen mentioned. In assessing theeffectiveness of utilization control proedures it ispossible to overestimate savings, and be unawareof shifts in the cost of care. Wyszewianski and I,have indicated asome of the ways in which this mayhappen)" A *port by Ike Lnititute of Medicineprovides a, good surnmarr l':ictors.to be consid-ered are that. the days of care saved nA be lesscostly than the average cost per day. that capitalcosts are nut reduced proportionately to variable,costs, that the hospital may function.inefficiently ifbeds remain 'empty, that the physician may not be

as productive in caring for some patients outsidethe hospital, that expenditures for carei)givenoutside the hospital in place of hospital care willrise, that these expenditures may not be coveredby insurance, and that nonadmission to the hb4pi-tal or premature discharge may generate futurecosts.

Specification and sting of system-design ele-ments

Several activities al ady described fit under this

new category of rese h and development pro-jects. These include th design of alternativecriteria formats and record s tems. Selected addi-tional features of system desig will be described

below.

Specifying the appropriate cut-oir points instandards The determination of the appropriatecut-off points or levels in the standards formonitoring is a critical design element becauseboth the yield and the cost of the monitoring ef-fort are heavily.influenced by that. There may als,o

be other consequences, for example to the socialacceptability of.the system and dysfunctional adap-tive reponses to it. An excellent example is the de-terfnination of the most appropriate "check,points" for recertification of further hospital stay.To make this determination it is necessary tospecify.the factors that. go into the analysis, toidentify what information is necessary, and to im-plement the analysis, first, in model form and,later, in practice. Wyszewianski and I have indi-cated one possible way to proceed." Averill andMcMahon have offered a mathematically morerigorous model."' I consider the further de-

Lelopsnent and 'lest' of these proposals as mat-ters of high priorit .

1

Sampling and "enrichment" techniques There

are two aspects of this subject that tend to overlapand become confused. The first has to do with thekind of probability sampling designed to obtain anunbiased picture of a specified universe ofphendmena. Obviously, this is a critical issue inmany assessment efforts; and much work isneeded to develop efficient means of stratification'and sampling. Some studies mentioned earlier,bearing on the heterogeneity and homogeneity ofperformance and the factors that influence it,Would contribute to the knowledge needed tosample more efficiently.

There is a ther kind of selecticy which is notsampling in the/ statistical sense, but a method ofscreening. its Intent is to increase the yield formonitoring: to hit pay dirt, so to speak. Ideally, amethod Is wanted that would pick up every casethat is managed suboptimally, while it excludesevery case that is managed at an acceptable level ofquality. In other words, a screen is wanted that is100 percent sensitive and 100 percent specific. Inthe real world We, obviously, have to settle forsomething considerably short of this.

In considering,further research in this area, letme begin by pointing out.that not enough' atten-tion has been given to whether total coverage isnecessary if a monitoring system is either to give afair representation of performance or to hie effec-:

tive in achieving improvement in performance.Samples could be no less effective in achievingboth these objectives. Pilot studies to verify thispossibility would rank high in my list of priorities.

A fair amount of work has been done on de-veloping selection or screening methods that areintended either to increase the yield from monitor-ing or, also, to bring about the most efficient sep-aration of questionable from non-questionablecases. Examples are the work of Wolfe,'" Riedel 0al.,"s Rubin,60 and Glass et al."' Brauer brieflydescribes a variety of selection mechanism-1 and

gives a longer account of an "adaptation of themethod described by Riedel et al, to rev,iewing psy-chiatric care.'" Certain procedu-res that begin theprocess of review after observation of poor out-comes, such as the methods used by Williamson"and by the PEP, system,'° can also be regarded asconcentration, enrichment or screeningmechanisms. Despite all this, relatively little isknown about the effectiveness of such screeningtechniques as measured by sensitivity and specif-icity ratios. Mushlin does provtide a test of hisscheme, which depends on rev4wing the recordsof Those whose original symptoms have not im-proved within a month of reporting for care. For

26

and specificity of this method are quite impreslsive.. By contrast, the method advocated by Riedelet al., which is essentially the selection of statisti-

f ally deviant cases, d s not seem to perform verywell,: ipdging by thehave seen."

One particular method of selection, that in-., volved in the "tracer" methild developed by Kess-

ner et al.,12 ." has been described in an earliersection,

inary reports that I

Larger elements of design In this section, I wantto call attention to the design of the assessmentand monitoring, endeavor as a total system inwhich there is a mutually supportive functional re-lationship among parts. I have tried to developthis idea to ap earlier work.'" Various expressionsof it are daily seen,, in the design of a variety of

. . systems including that of the PSRO program. Theconceptual apparatus and the methods-for testingsuch constructs will probably come from systemsen sneering and analysis, and will embrace bothtec nical and social phenomena. Since systems

\virialysis is another of the many subjects aboutwhich I know next to nothing, all I can do is toexpress the hope that it has something to contrib-ute. If it does,not, the work of designing andtesting these larger systems s ill not stop, but theunderlying principles hat govern design and ef-fectiveness will have to be formulated as the,/ con1ruction goes on.

Comparative studies of quality using differentapproaches

A great deal of insight into alternative methodsof assessment can betibtained when they areapplied to the same set of records, and the result-ing estimates of quality are compared. Anexcellent model is the early work of Brook,"4,par-ticularly when studied in its more dttailed

More recently, Brook et 1. have -pro-posed that separate sets of process and outcomecriteria be developed simultaneously for a numberof conditions, with subsequent comparison of theratings of quality accorded to the same care usingboth sets." The primary purpose in such studies isnot to find new facts ,about' the link betweenprocess and outcome, but, given existing knowl-edge, to determine.whether the proper formula-tions of process and outcome criteria have beenmade. However, when discrepancies between rat-ing based on the two sets are found, they couldlead to questions about the validity of what wasthought to be known about the process-outcomerelationship.

proaches

Irwrecent years, a number Of promising new ap-proaches have been developed and tested more orless rigorously. In most cases, a great deal remainsto be done. To select one or more of these ap-proathes and invest in, further. testing could bevery rewarding, since it would, build on existingwork and benefit from''the advice andrcollabora-tion (and, sometimes, the afterthdughts)___of theiroriginators. In fact, many of the proposals for re-searth that I have included in this review are de-rived from a critical examination of what has beenpublished about these new approaches.

At the .riskor some repftition, let me mentionsome of the approaches that I think are mostpromising.4'he order in which they appear is notintended to signify either perceived impottance orpersonal preference.

Outcomes as measures of quality. The major re,cent exemplar of this approach, the InstitutionalDifferences Study, demonstrates both its useful-ness and its limitations.72 The fundamental as-sumption in this and similar, studies is that whenstatistical corrections are made for known risk fac-tors that influence outcome,a great deal, if not,all,of the variation that remains is accounted for bydifferences in the quality of care. I believe that adirect test of this assumption should be attemptedby independent assessment of the process of care,using a variety of methods. I suspect that the re-sulting' correlation between process and outcomewould be low, suggesting .the need Tor bettermethods to standardize for risk factors, as well as hmore fundamental approach to process' assess-ment. In particular, the decision to operate or notwas not subjected to assessment in.the InstitutionalDifferences Study, as the investigators take painsto point out. I suspect that many of its anomalousfindings are traceable to this basic weakness.

The method of assessing proximate and inter-mediate outcomes of care, so well described 'byBrook et al.," is another promising approach that'requires and deserves much further developmentalong the lines indicated by its proponents. Thisincludes extension to other diagnostic categoriesor conditions; actual implementation of thecriteria already developed to test feasibility, accu-racy and cost; and comparison with simultaneousprocess assessments.

Outcomes as cues and motivators for process as-sessment There is no sharp line of demarcation be-tween the use of outcomes as measures of qualityin their own right and as screening devices to'select cases for process review. In fact, my attribu-

2?

20

don of primarily one unctiop to some methodsand another function to ther methods is likely tobe. challenged by their o iginators and advocates.However, I do perceive t e approach developed byWilliamson" and used by Mushlin,9 as well assome basic- elements i the PEP method of theJoint Commission on Accreditation of Hospitals,10to fall more comf itably in the category ofmonitors of outco rather than assessments ofquality based o'n o tcomes. The method describedby Brook et al." appears to be intermediate be-tween the puree forms of the two classificatorycategories that have proposed. But none of thisdiscussion on cl ssification need deter usnfrom not-ing the potenti I usefulness of these methods andinvesting effor in their further development. I amparticularly mpreised by the simplicity ofMushlin's ap roach, and its great success inseparating sign ficantly deficient care from accept-able care. Whe her this will remain true when ad-ditional conditi ns are tested remains to be seen.As to the PEP pproach, it offers many opportuni-ties for further development,, for example the de-termination of esponsibility for complications thatoccur in -the h spital and the. inclusion of sonicoutcomes that appear after discharge from thehospital, as pr posed many years ago by Cod-man." To retu n to the distinction which formsthe basis of my Classification, if these methods toindeed use outcOmes as monitors and screeningdevices, rather than as more complete representa-tions of quality: a dear recognition of this distinc-tion could lead to simplification of the measures ofoutcome and to their assessment in terns of theirscreening effidency. I believe that this Would be a

very useful and important t4velopment.

The occcurence of preventable adverse eventsRutstein et al. have recently reminded us,of thepotential usefulness of this, time-honored methodof monitoring the health care of a population." In

this Method, attention focuses on, outcomes andother events that are preventable, at least to a sig-nificant degree, when good care is available and isused. Obviously, this is little different, in principle,from the approaches described in the precedingsection, except that-we are now speaking of popu-lations rather than of patients. A historically signif-icant method of quality assessment and controlthat probably helongs under this heading is repre-sented by the activities of the maternal mortalityand perinatal mortality committees originating inthe landmark studies of the New York Academy ofMedicine.""" Since then, much information ofthis kind has been accumulated over long periodsof time in may states and localitiet. The useful-ness of this historical material is demonstrated,by a

recent analysis of Maternal mortality data inMichigan.11 A current application using infant

deaths in hospital, has been reported recentlyfrothiEngland."° The key feature.Of these studiesis that the problerp of attribution is handled by acase-by-case analysis that leads, to a determination..of whether there were preventable or avoidableadverse circumstances and by assigning responsi-bility for these. Broader application and testing ofthis method, using a wider range of conditions,seems to be well worthwhile.

The occurrence of preventable progression ofillness or disability This is another member of thefamily of outcome-oriented methods which cannotbe fully differentiated,from some of its compan-ions, except in emphasis. It has particular affinityto the preceding category, Preventable AdverseEvents, except that, in this instance, the focusthe preventable progression of illness from anearlier stage, which is presumably more amenableto treatment, to its later, more advanced andrecalcitrant manifestations. This approach owes itsmore recent saliency to the work of Gonnella andhis associates who refer to it as the "StagingConcept."129.1" In one way, the \staging of disease

creates more homogeneous categories, so that theattainment of outcomes can be compared withgreater confidence that differences are attributa-ble to care. However, Gonnella -et al. also arguethat the stage at which a disease comes under care,either initially or at some later date, telts ussomething about earlier ,access to care 'and thequality of that care, if care is Provided. As in alloutcome studies, the problems of interpretationare many, but this approach does simplify popula-tion monitoring to some extent.because the neces-sary data can be obtained from within the patientpopulationjor example, a hospital may be unable

to precisely identify the population it serves or toobtain useful.. population data, 'but it can ,charic-terize all, or,a sample of, its admissions as to stageof illness and preventability of progression to sucha stage. Further empirical testing is warranted.

Indicator conditions:. "trajectories" and "trac-ers" There is a large number of studies in whichone or more conditions are selected and the careerof patients with these conditions is followed as thepatients proceed through, the System. This couldbe call the "trajectory" approach, since the em-phasis is on what happens at each successive stepin a progression that is, too often, a tragic Odysseyof accumulated failures. Examples are provided byBrook and Stevenson,'" Starfield and Scheff,132Novick et al. "4 and Shorr and Nutting." The"tracer" method developed by Kessner et al. canbe seen as a highly systematized selection of suchindicator conditions, each with its distinctivetrajectory. The systematizing or organizing device

28

s

is a prior conceptual mapink of a field and the. purposive selection of conditions "lo represent all

the major elements in that fie41.9313 Anothercharacteristic of the "tracer" method is its em-phasis on combining population and patient &lullto achieve an epidemiologic investigation of theproblems of medical care. However, thisepidemiological perspective could be incorporated'in the "trajectory" approach, in which case eachtrajectory becomes a tracer (more accurately, thepath of a tracer). Such semantic. games aside, thereis much opportunity for further work in this area. Iparticularly like the notion of a planned selection of

% tracers, with, a view to systematically, sounding thecorpus of medical care-in its totality and to includeboth patients and non-patients. However, since thisrequires, a massive . effort, more modest and cir-cumscribed applications should be tested first.

Second surgical opinion programs An idea thathas captured a wide audience is the possibility ofcontrolling unnecessary surgery through eithermaking available or requiring second surgicalofiinions.133 This is a particularly interesting ap-proach since it is truly preventive. There are manyopportunities for research and development here,including tests of the reliability and validity ofmultiple surgical opinicins, acceptance by clientsunder voluntary and mandatory systems, accept-ance by physicians, effect on relationships amongphysicians and surgeons in a community, effect oninitiation of recoinmendations for surgery, effecton the client-physician relationship, and so on. Toanswer the question of validity:long termfollow-up under, controlled conditions would benecessary. Stich studies are now in progress.'"

Computer-aided management I hive discussedearlier the affinities between computer-aidedmanagement and several facets of quality assess-ment, including .the design of records, the identifi-cation of critical events, and concurrentimonitor-

. ing. I. see this as an area of much fruitful furtherdevelopment I

Integrative measures of quality

I have already said that most studies of quality,focus on a relatively small segment of care pro-vided by one professional, and, that there is needfor more "integrative," measures that include thecontributions of several professionals during corn-.,

plete episodes of care and sequences of suchepisoldes. I have also pointed out that outcomemeasures, especially those that are more inclusivein extent and duration, are by their very nature,

integrative, The approaches that Use'''the "trajec-tory" or "tracer" methods also have an integrativeproperty since they often include outcome as wellas process measures, and follow the course of carethrinigh successive stages, levels and sites, so that ..

they reflect the cinnulation yf deficiencies at eachof these junctures.

There k need to develop methods that incorpo-rate process and outcome elements that are expe-chilly selected to represent the contributions of theseveral professions that are involved- in patientcare, because these elements are particularly .re-sponlive to the contributions of each of theseseveral professions.

Another way of taking a more complete orintegrated look t performance is to use as a unitof analysis thee tire case load of a practitioner orinstitutional prov er, so that the assessment in-eludes not only th adequacy of care, but also theoptimal allocation o esources'among cases.

Applications to special area*

On the one hand, there is need to developintegrative-veasures of the quality of care. On theother hand, `there is need to adapt assessmentmethods to special populations, to categories ofcare, to particular professions, and so on. This isespecially true when an agency or organizationthat provides patient care includes very large ele-ments of very diverse programs, including generalambulatory and inpatient care, long term care,physical rehabilitation, care for mental illness and

-alcoholism, etc. Within each of these categories,care could also be differentiated according to pro-fession, for example: pharmaceutical services, den-tal services, nursing caresocial work, dietetics,physical therapy, radiology, pathology, anos-thesiology, and so on. It is impdssible for any onereviewer to encompass the literature in all theseTecialized areas. I shall not even try to offer 'apartial description. However, the reader seepingan introduction might find useful some selecteditems that have come to my attention.

Freeborn and Greenlick have attempted to sys-tematize approaches to the assessment of ambulat-ory care.'35 Christoffel and Lowenthal have re-viewed the newer methods that are available.'35 Apublication of the American Society of InternalMedicine provides a recent anthology on ambulat-ory care assessment."' The American Nurses As-sociation has performed a similar, function formethods of assessing nursing quality.'" A collec-tion' of papers in the cfuality of pharmaceuticalservices can serve as an introduction to that very 'interesting and 'important area of application."9

re will be found in. a publication by Knapp andt .140 A collection of papers on the assessment

of al care, though not so recent, could also be .

22

)helpful,"1 Much information about the assuimentof mental health services' in genlorwl, together with

detailecttinformation ab6ist one approach,. will be

found in, a book by Riedereval.'"i.

Consumer perspectivis and the consumer's roleI

There are many who are distressed by the neartotal- domination that.. the professionals have exer-cised over quality assessment, fro its deepestroots to its most slender branches. nd yet, it hasbeen very difficult Ito involve the consumer in

quality assessment in a meaningful way. Aware of

this problein, The Institute of Medicine has iden-tified consumer participation as one of five"priority areas for quality assUrance," and hasdevoted considerable space to it in-its "researchagenda." " I' In this" section. I shall deal with some

, ways in which consumers can participate in quality

assessment, hoping that I will stimulate furtherthought, research and.adevelopment. ,

Representing client. values in the definition ofquality At' the root of quality assessment is a view

of what attributes of care constitute quality, whichimprints itself. on evernhing that follows. ft is,therefore, important, as 'I have argued earker, that

the clients' values, preferences and expectfions beincluded in the definition. of quality from the very

first. ;The choice of outcomes as a measure of quality

is, in itself, a means for making the notion of qual-ity easier for the patient to understand, and closerto his concerns. But not all outcomes are equallycomprehensible or relevant. Outcomei that are de-fined in terms of physical or, social function aremuch more meaningful than clinical, physiological

or chemical measurements. The iraluations that are

placed on alternative outcomes, when there is achoice, could differ not only between clients and

professional's, as groups, but also among clients asindividuals. All these speculations are subject toempirical study, as is the degree to which the con-duct of care takes account of client values andpreferences, collectively or as individuals.

, As we said earlier, clients also have preferenceswith respect to the process of care. Generally,these focus on, the management of the interper--sonal relationship. One cannot emphasize too

- strongly that in this one aspect of care the patientis fully as expert as the professional, if not moreso. .In fact, a good case could be made for havingthe client set the criteria of good care in thisretard.- Clients also have views about the-technicalcomponent of the process of care. Sometimes,'these are of debatable validity, as when an injec-don is demanded or, even, a hysteiectomy: At

other times, the expectations of the well-informedclient can be quite valid, for, example when morecomplete prenatal care is demanded or too ready

use of some x-ray examinations or 'of antibiotics is

resisted. In any event, it would be interesting tostudy the Influence of such expectations on pro-fessional performance.

I have already commented on the dual nature of

client satisfaction: aran "outcome" of care, and as

a judgment 'on care. In the latter instance, it could

be seen as the patient's estimate of the quality of

.care. It is remarkable how infrequently the-pa-tients' estimate of the quality of care he hasreceived in a defined instance has been compared

to that of a professional estimate of the, quality of

the same care. The only example that I can think

of was reported by Ehrlich et al. many years

ago."' This is an obvious area for further study.

Clients as sources of information For some dataused in assessment the patient is the only .au-

thoritative source: for example, data onanda

pa-

tient's knowledge, opinions, satisfaction nd thelike. For other data, the client is an alternative orverifying source: for example; concerning services

received and assessments of function or disability.

There is .much rooms for research on ways ofobtaining reliable and valid information at lowcost, and its incorporation into quality, assessmentand monitoring activities.

It is perhaps ironical that "the patient's'record,"

in any medical setting, is not only totally barred to

the patient, but also.cdntains no 'direct entries. by,

the patient concerning how he feels, what he

knows, or how he perceives.his care. It ould be afascinating experiment to see whethe it would bepossible to have the patient tries into therecords, eitherin narrative form tor'as checkmarks

on a list of questions. It has been suggested thatpatients might be unwilling to express- negativefeelings for .fear of reprisals. This is subject totesting and, if verified, entries might be made, on a

separate document that only later becomes part ofthe record. Even the answers to the iluestion,"What was the one best'thing that happened to youtoday?"would be most revealing.

Participation in monitoring Participation inmonitoring can perhaps occur indirectly. and in-formally through the manner in which the wellinformed patient responds to physician initiativesand participates in the client-practitioner interac-

tion. This is a matter that can be studied.A specific example of this more general category

is the degree to which patients participate in sec-

and surgical opinion programs and how they react

to non-copfirrhation of the initial reCominenda=tion. There is alt(eidy evidence that participation

,

In voluntary programs is not high and that patientsdo not always act in accsrd .with a second recom-mendation, whether at is confirmatory of the firstor not.18$0" 4ttempts could be made to alter client'behavior and to test their success.

Some have argued that the paienti medicalrecord should be accessible to him at all times.Shenkin a d Warne have proposed that thisshould be required by law, and have speculated on''the possible advantages and disadvantages thatmight ensue.'" Stevens et al. described experience

with- an actual trier phiast.ggests that at leastsome patients can ',participate usefully in monitor-

; ing their own care, while others either catinot orare, unwilling to try.,'" More work is cal for.

It, is not clear, to What extent patients canparticipate in a more structured and formal man-ner in. monitori the care they themselves receive,but there are many opportunities to find out. Forexample, Bouchard et, al.'" and Burger"' havereported on experience with involving the patientin auditing' the problem-oriented record-by havingthe physician's evaluation sent to the patient forcomment.

7:Membership in audit and utilization reviewcommittees As far as I know, the inclusion ofclienis on audit and utilization review committeesis virtually unexplored territory. Goldblatt et al.describe professional reailance to inclusion ofconsumers on such committees in a project onmental' health evaluation.'" But, they also notevery briefly that a "consumer opinion subcommit-tee" that :tindeperidently investigated clients'opinions about care and analyzed complaints areferring institutions and practitioners" influencesselection of topici as well as cases for review. Inthis way, "consumer dissatisfaction could betutVed into criteria .for good care."'" In the samesway, consumer participation on grievance corn-mittees'and similar bodies could be linked to theactivities 'of quality assessment. There is urgentneed for careful trials of a variety of mechanismsfor involving consumers directly and indirectly inquality assessment and assurance.

Patient contributions to care. The importance ofthe client-practitioner relationship,to qualitysessment has been a recurring theme in this paper:The care bf the patient is a joint enterprise whichincludes contributions by the patient as well as theprofessional. Quality assessments of care may givethe professional too much credit for success or toomuch blame for :failure unless careful attention isgiVen to the role of the patient. Btu, there is much

; more involved than simply deciding who is respon-sible for what. The assessment of the patient'scontribution to his _own .care, and. study of the

factors that influence that contribution, is a legiti-mate and important. area of research in its 'ownright.

Quality assessment and monitoring as a social,prooess

Perhaps the most difficult problem's in estab-lishing quality and utilization control mechanisms,and in operating them effectively, are social ratherthan technical. It is of the utmost importance toredress the imbalance in past and current researchby paying at least equal attention to qualitymonitoring and control as a social process. In my 23.earlier work, I have described briefly the kinds offactors that provide a context of quality assessmentand monitoring.'" I have also speculated on thenature of the factors that irlfluente the effective-ness of monitor\ng and review.'" More recently,Freidson has presented an incisive analysis of thesocial processin the implementation of PSROs "aspart of a larger class of issues connected with thesocial psychology of work and its control."'5° Ad-ditional speculations, from a more operationalviewpoint, can be found in a series of comments pna paper by Morehead,'5' in a paper byBellin.'5° Jacobs et al. describe and discuA factorsin the implementation of the PEP system of as-sessment developed under the auspices of theJoint Commission on Accreditation of Hospitals.",Goldblatt 'et al. give an excellent account of ex-perience with utilization review committees .in acollaborative project in mental' health evalua-tion'4° Using these 'sources, and others like them,it would not be difficult to formulate a series ofhypotheses as a starting point for research. Butthere remains a great and urgent need for anapproach to the study of the implementation of .quality monitoring that rests on a sound, andsystematic conceptual base. The development ofthis conceptual foundation is, itself, an area ofscholarly research. ,However, it is likely that nounifying framework will emerge, and that thissocial phenomenon, like all.others, can be seen andunderstood using a variety of theoretical con-structs that are provided by economies, politicalscience, sociology, organizational theory, an-thropology, history, law, and so on. My pessimiimnotwithstanding, the student of medical care or-ganization will 'continue to hope that in some way,relevant contributions from the theoretical trea-sure of all these disciplines can be brought to-gether coherently, so he can understand-fully theproblems that he faces.

At the broadest level, there is need to under-stand what might be called the politics of qualitymonitoring and control. This could include 'therole of government, of the, organized professions,of the organized constituencies, including the

31

24

health insurance sector. The interrelationships be-tween regulatory government agencies and the or-

ganized profession is a particularly critical area for

study. All these forces need to he understood asthey interact at national, state and local levels.

At the level of thethospital, :mil similar idstituttions, it is important. to understand the- powerrelationships Within the hospital as well as those.between the hospital and, its environment. Theorganization of physicians in the immediate emit-,ioninent of the holipital and .within it is a particu-larly critical element. Comparative studies ofhospitals that have different goals and/or aredifferently organized could be quite revealing.Such studies might include comparisons ofgovernment-owned to community hospitals, and of

hospitals that emphasize teaching and research tothose that confine themselves to patient care.

The local PSRO functions as' a key link between

the hospital and the organized profession in, itsenvironment, and between the latter and gov-ernmental ,regulatory and financing agencies, as

well as other third party 'payers. The dynamics ofPSRO operations should Jiecome, therefore, anobject of intense scholarly scrutiny. Unfortunately,it is quite likely that this will be countered by anequally intense determination to avoid beingstudied. Nevertheless, careful study of the PSRO

as a social organization is absolutely necessary, and

openess to such study should be a condition for

formal recognition.In this progression from larger to stnaller,social

units the next critical-level is that of the audit andutilization review committees, where much of work

of assessment and monitoring is done.-The struc-ture and roles of these committees should be

studied in. differently. organized hospitals. withdiffering linkages to the local PSRO. Thedynamics and effectiveness of the committees arelikely to be influenCed by the structure of thetommktees themselves, including variations inWho is represented on the thembership.roster, and

how the committee' is linked, structurally andfunctionally, to centers of power and inflionce inthe hospital as a whole. ran earlier section,mentioned the, importafce of studying thedynamics of developing thte criteria and standard's

that ,operationally define quality.At the most disaggregate level of analysis one

finds, as always, individuals who hold key rolesiand who carry on their shoulders, as it were, theburden of the, most massive of social enterprises.Depending on whether these persons are seen as

prime movers or mere pawns, their behavioreither determines. or reflects the manner in which

the total enterprise ultimately functions. In thisinstance, I see as if at the focal point of some giintlens turned to' the sun, the lonely figures of the"nurse coordinator" and medical adviser." If I had

a choice, I would go first to these, and try to un-

derstand how they work With each other, and how

each rikated structurally and functionally to theirrespect peer groups, to the internal utilizationand audit ,cominittees, the cliriical departmentchiefs, the chief of staff, and the 4ospital adminis-trator. There are usually others who are important

as leaidets:Ahough informally; and still others who,,

like the pathologist, can have disproportionate in7fluence because they control infotmation critical tothe assessment and, its conclusionii.

And, finally, what of the individual physiciansthemselves, who are the object of this seeMinglyunrelenting scrutiny? The manner in which theyrespond individually, through/ their formal and in-format] groupings within the Aospital, and throughtheir many-tiered and interilocking- professionalogranizations, including 'the PSRO, determineswhether the program achieves its objectives, orwhckher it is so watered down and subverted that it

becomes a ponderous,and costly apparatus, 'mak.-

ing i brave shOw, but achieving little.

Effalveness and the factors that Influence it

Finally there is no way of escaping, the most

Momentous of all questions: whether vality and

utilization review .activities are effective, and suffi-

ciently .so, to justify their immense social and

monetary cost. It is remarkable that even now,when we have made a political decision to con-struct the awesome machinery or the PSRO, theanswer is that we do nor know. ome'"fyears. ago I

reviewed, what was' then known about the effec-tiveness and costs of 'quality and utilization reviewmechanisms.123 Very recen , a committee of the,/Institute of Medicine reasse sed the situation notonly by reviewing the lite tune but also by ob-

taining information directly from operating pro;grams.'" My conclusion was that hi some instancethere was success, whereas in others there 14asfailure, and, that we did know for fertain whataccounted for either, though one could speculate

at length on the matter. I understand the cdo0u-sitaus of the Institute of Medicine to be not muchcf ferent. It is still, not clear what hospitaLmedical'au its atcomplish, if anything. by contrast, theut ztion control programs of hosprtals occa-

sionally report savings, but these tend to ver-,

estimated because of improper account as-

sumptions. Ambulatory care claims review, wherestudied, has been cost effective, but this is mainly

or entirely 'due to the administrative component,as distinct from .

professional peer eyiew. All these"savings," when they do occur, are 'to the fiscalintermediaries. The social costs' can be shifted. As

to the effect on the health of people, almostnothing can be said.

There are many reasons for this rather dismalstate of affairs. The most fundamental are the

O.% .absence of sound conceptualization of the problemto be addressed, the deficiencies in the basic toolsof measurement, and the absence of soundly de-signed and properly controlled studies. No ac-cumul don of case reports from operating agen-cies, : I) matter how lengthy, can resolve theseproblems. Case studies do, however,'have the vir-tue of generating hypotheses for further testing.

In subsequent sections. I will suggest some areasfor study, without,trying to cover the crone rangeof possible research. It will become obvious to thereader that there is much nverlaiihetween theseproposals anti others that were made earlier 'underdifferent headings. In particular. there is a closetie between the specific considerations that comeup in this section .and the issues thaqt were verybroadly sketched in the preceding section on Qual-ity Asserent and Monitoring as a Social Process.

Changes in physician and client behavior It isimportant to document the changes that occur inthe behavior of physicians and other practitionersas a result of instituting quality and utilization re-view mechanisms. Rather simple before-afterstudies are useful: but, where possible, contem-poraneous controls should be provided. In thesestudies, it is necessary to keep in mind, and lookfor certain "dysfunctional" behaviors, such as atendency to lengthen stay up to the "checkpoint,"or to discharge prematurely, as a response to a cer-tification program: the. likelihood that physicianswill "manage by criteria," resulting in many re-

' dundant procedures: and the possibility that eva-sive actions will be take, for example by using adifferent diagnostic nomenclature or by movingpatients to other settings. It is safe to say that ev-erything that human ingenuity earti devise will beused to tame a regulatory mechanism, and the re-searcher must be prepared to anticipate and studysuch behavior. Some attention to possible adverseor unintended effects can be seen in an interestingpaper by Brian on the impact of a utilization con-trol program -in California.'" Brian concludes thatthe 'program -was effective 'without evidence thatneeded care was denied or costs shifted to others.In direct contrast, two reasonably well controlledstudies of a hospital-stay. recertification program

.in Pennsylvania showed no effect on hospital useeven though the state,Medicaid agency was muchimpressed by the reduced rate of unjustifiedstays.'".'," A reasonable, though unverified, ex-planation 0.1-this discrepancy is that the programdid not alter hospital use, but did improve tabdocumentation' needed to obtain, payment farcare.=s' If so, it is only proper to ask whether this'new documentation is a better representation ofthe truth, or only a more credible distortion. Tothe extent that it is the latter, we may, a% a society,be turning o t ?he mo expensive fiction every,st

written!

1 included the client in the title of this s ctionbecause of a conviction that any change in t eliv-havior of the practitioner is likely to haveterpart in the client. Almost no considerao n hasbeen given to client behavjor in studying the offeels of quality and utilization control mechanisms,and I am not sure that there will he any, unlessconsiderable costs are shifted to the client.Nevertheless it is interesting to speculate whetherpatients may be morekely tq patronize physicianswho are less scrupulous in accepting the strictureson hospital admissions and length of stay. MightThe patient develop symptoms when informed thatthe approved hospital stay is about to expire?Could the local hospital lose community supportfor seeming to repeatedly refuse admission or tothrow patients out?

Effect of technical design Characteristics I have'discussed in an earlier section some technical de-sign elements that might influence the perform-ance of the quality monitoring system. The rela-tionships to effectiveness could be inferred fromobservations of existing variants or tested by inten-tional manipulation under Controlled conditions.Examples include studies of the effects, of differ-ent criteria formats, of varying the hospital stayrecertification checkpoints,. and of testing thecost-to-yield ratio of alternative sampling and en-richment schemes..

Intra-institutional "social design" characteristicsThis category subsumes a very large area much ofwhich I cannot see clearly. .1 will mention only a'few of the more obvious kinds of studies.

It seems to me that the legitimacy of the qualitymonitoring effort in the hospital, and the degreeof commitment to it by its key figures would be avery important factor in the effectiveness' of thiseffort. The- written and verbal declarations ofboard members,Administrators_and physicianleaders would be one source of direct information.Perhaps more valid would be the inferences drawnfrom how the assessment and monitoring effcirt isstructured and how it functions. Much can belearned from an examination Of who chairs theaudit and utilization review committees, and whothe members are. It is also important to know"whatdecision-making power the committees have, howtheir recommendations reach the executive eche-lon in the hospital, and the degree of influence thecommittees have on..tlie key centers of power inthe hospital.

The. legitimacy of the criteria and standardincorporated in the monitoring system could be aparticularly important factor. One significant 'vari-able is whether the criteria are externally imposed,developed internally or a mix of the two. If the

33

25

26

, .'crite?ia aee, at least to some extent, internallydeveloped, the degVe of participation in criteria

1 formulation may influence the adherence to. thecriteria by the physicians as a whole, or may dif-ferentiate between participants and nonparticip-ants with regard to adherence. If the criteria areexternally sponsored. the identity of the sponsorcould be important: for example, whether it is aninsurance carrier, a government agency, the local 1

PS110, the Joint Commission of Accreditation ofHospital's, the American Medical Association orone of the specialty societies. Ina large systemsuch as the VA. sponsorship by the central officeversus the local institution.could be a differentiat-ing variable! .

e structure of' sanctions and incentives as it401 ICS on individual practitioners cannot fail tohave an important effect. One le.atue is whetherthe monitoring system confines itself to studies ofpatterns of care or whether it goes beyond that toidentify individuals whose practice is called intoquestion. 'I'he methods used to communicate andinterpret findings are determined to a large extentby' the initial decision to identify or snit identifyindividuals; but, irrespective of that, there aremany options. Nelvsletters., general staff meetings,

( and meetings in smaller departmental or sub-departmental units could have different impacts.It is also itriportat who is responsible for com -municating the information:. the lialtire of hisinvolvement; and whether that person meets withthe entire. Staff, a small group, or it single indi-

--' vitlual. r-

More important still is the manner in whichfindings about performance become instrumentalin influencing the careers of individual practition-ers. fit may be that the major consequence. ofnonadherence to standards is 'approval or disap-prov- 1 of payment by an insurance carrier org ernment program. While important, occasional'

brushes with a third party payer may not be as

effective in influenscing behavior as would be thecertain prospect that the physician's performancerecord will be considered in determining his prac-tice privileges, promotions, access to prestigiousappointments, and other organizational .rewards.The use of . rewards and their possible impactshoulcUreceive serious attention. Would it be pos.sible, for -example, to grant priority in admiSsion to

the patients of physicians Oho have ,a consistentrecord*;4, very few unnecessary admissions orstays? While thiy'absence of a reWard'is, itself, apunishment it could well be that thereis a signi&cant difference between a system that focuses onpunishing wrongdoers and anotlier that stresses

' .0rewards to (hose who have an excellent record ofperformance.' Finally, any incentive system will be

inoperative if knowledge about it is not shared andif its. certain and impartial implementation is inquestion.

/'Pour performance is not always primarily at-

tributable to individual failure. Quite frequentlythere are organisational problems that 'interferewith good work by practitioners. In that case, andoften when individuals ate at [atilt. it is necessaryto make changes in the organizatiiln. This requiresthe full support of those who hold executivepower. Ilence, the manner in which the qualitymonitoring system is linked to the executive, and '

the mit ore of its influence at that level, becomecritical elements in studying the factors that mod-ifyperfoance.

Education, either alone or in conjunction withsanctions. features prominently in attempts tobring about clinge in the behavior of practition-ers. The variables that are likely to influence -elec-tiveness include, first, the relative emphasis placed

on education as compared to sanctions. Thep,there are different educational strategies thatcould be more or less effective. On distinctionthat has been the.subject of much sp Illation isthat between an educational program b ed ontopics of general interest and on that is' ed by

'audit results. The latter approa can' be highlyindividualized by tailoring continuing education tothe deficiencies in an'individual's performance..Itis also claimed that participation in audit and re-view activities is itself educatilmal and helps*moti-vate change in behavior. All these speculations, aswell as other hypotheses about the differential ef-fectiveness of alternative' educational strategies,are open to testing.

Strategies of client education are also a relevantvariable, since patient cooperation may be an im-portant factor in achieving the objectives of qualityand utilization review.

Supra-institutional "social design" characteris-tics The sanctions and incentives inherent in ,aquality monitoring and control system may act onthe institution as a whole, in addition to 'theireffect on individual practitioners. The interests ofthe institution are likely td have great impact onhow committed it is, as a collectivity. to quality andutilization control. For example; it is generallybelieved that when beds are plentiful, neither theinstitution nor individual physicians are motivated

to keep patients out. Self-interest works in pre-cisely the opposite direction_ to .the objectives ofutilization control. The same; is true ifr budgetallocations to an institution depend ,on occupancylevels, and if savings from more prudent manage-

.ment cannot be retained by the institution. Insome situations, it may be possible to vary suchfactors under reasonably controlled conditions andto study the consequences.

The nature and extent of participation inmonitoring and control by an external agency isprobably a critical factor in effectivenef. Such an

.3 4

external agency might 'be an insurance carrier, agoveunnent agency, or a PSRO. In a system suaiair that of the VA, the central Office probably.Oeursa paaallel relationship to the local hospital. 'Therelationship ,Between the external ageiicy and heintra..institutional quality monitoring apparakuscan be structured in a variety, of vfays, and theconsequences examined. For example. under theSRO program, the hospital may be delegated,partially delegated or nondelegated with respect toquality and utilization control activities. ThePSRO, or the analogous external agency, could-Obtain repons from the institution undersupervision, duplicate certain review activities, in-dependently-obtain data additional to those hvail-able to each institution, analyze and interpret data,engage in consultatory and educational activitifs,and so on. It may even be feasible to have hospitalstake turns reviewing each other. The effects 61 ).each of these activities, and of the various ways in(which each can be implemented, are legitiniateend important objects of research.

a Social- organisational characteristics and medical'technology Based oh more general work in or-, ganizational behavior, Scott et al. havehypothesized that a given organizational form maybe more or less effective in influencing perform-ance depending on the technological characteris-tics of the work being done." Somewhat along thesame line, though independent of the work ofScott et al., Wyszewianski has hypothesized thatthe effectiveness of a specific quality control pro-cedure is influenced, among other things, by thedegree to which it' fits, speCified technologicalcharacteristics Of the task whose performance it isdesigned to influence.'" Further research alongthese lines may provide insights into the effective-ness of discrete quality control practices, whilemaking a contribution to organizational theory.

Case studies of successfil and unsuccessful pro-grams. There is lively debate about the usefulness-

ad

of case studies as a method for research. Thecontention that ease studies are ( aluable methodfor generating hypotheses is co tered by theargument that it is impossible to understand any-thing about real life situations unless they are'viewed and interpreted in preformed ways. Casestudies cannot gengate knowledge out of ignor-ance. However, giien some general conceptualframework that suggests to the investigator whereto look and what kinds of things to look for, casestudies, I believe; can nudge the mind into newformulations that draw on what was formerlylatent or only dimly perceived in, the investigator'sthought. Besides, the 'participants in the eventsthat are under study have their own views and

explanations of what these eveuts Mean. Theseare. in a way, fragments of theory and snippets ohypotheses; and in these bits and scraps the alerinvestigator tint)/ find the roue outlines of somnew insight.

It seems reasonable to begin the investigation byexamining' rply contrasting situations of brit- 'liant succe 4 a ect failure. The factors thatcontribut. V5neastatould be more oharply.diffOnt se' large contrasts: But, fur-ther .e ugh the examination of in-terniediat cl`hirvegis Follow. Ahis must lead to More.rigorobs testing of specific

hypotheses.'

\Cost.e(feetivtiness and ein 'this section, Ichanges in behthat Seem to enhance or deter such changes. But,ultimately, a determinalion has to be madewhether any given monitoring mechanism is wor-thwhile, and which among alternihtive mechanismsis to be preferred. This requires the comparisan atlept of costs and effects, and, preferably, of costsand benefits. For example,. a committee of theInstitute of Medicine has estimated. flat an exten-sion of PSRO activities to cover all inpatient andambulatory care would require a yearly expendi-ture of 11/4 billidn dollars."4 What do we get inreturn? .

The returns to a program of,monitoring qualityand utilization are partly nionetary savings due toreduction in "unnecessary"- care. These savingshave to be set .aginst the expenditures that mightresult from`care that is added in order to improvequality, but also due to redundant care masquerad-ing as "quality." I have already discussed brieflythe difficulties in measuring these savings and ex-penditures, and emphasized the need tticlearlyidentify their social incidence. When thatYs done,the cost of establishing and operating the monitor-ing apparatus must be carefully determined socan be set against the monetary balance of its con-sequences. But,Ino m4tter how careftl and accu-rate,the balance sheet of savings and expendituresis; a definitive judgment cannot be made unless itis possible to measure the impact on health. Themegsurement healih and the assigning of amoetary value to it has also been discussed in anearlier section. Seeing how difficult a task this is,we need not wonder that the effectiveness of somuch that is done in medical care and its Organiza-tion remains less than completely evaluated.

.

st-henefit studies So far,e dealt with documenting

and in identifying the factors

..35

e

a 1

28 The task ofo

eviewing the extent of currentignorance and indicating ways of remedying itcalls for an approach that the reader may findoverly critical of what has been accomplished: andof what can yet be done_ , in the world of actiqn. AsI said at the beginning, it would be foolish to arguethat all efforts to monitor quality must cease while

we seek certainty about a near-perfect solution. Onthe contrary, we must continpe to act based on

what we now believe to be reasonable and feasible.But, we also need to find out whether we havebeen corljece in acting as we have, and to learn how

to dO better in the future. I hope that this paperhas made a small contribution to that continuingquest.

36

e..

ti

1. Brook, R.H. and Allyson-Davies, A., QualityAssurance and Cost Control in Ambulatory Care.,Santa Monica: The Rand Corporation, July

-1977.

2. Tompkins, R.K.; Burner, D.C. and Cable,WI.. An Analysis of Cost-Effectivenins ofPhary,pgitis Management and AcuteRheumatic Fever Prevention." Annals ofInternal Medicine 86:481-492, April 1977.

3. Forsyth, R.A., "Selective Utilization of Clini-cal Diagnosis in Treatment of Pharyngitis."

. The Journal of Family Practice 2:173-177, June1975.

4. Lusted, L.B., Introduction to Medical Diagnosis.Springfield, Illinois: Charles C. Thomas,1968.

5. McNeil, B.J.; Keeler, E. and Adelstein, S.J.,"Primer on Certain Elements of Medical De-cision Making." ,New England. Journal ofMedicine 293; 211-215, July 31, 1975.

6. Raiffa, H., Decision Analysis: Introductory Ledtures on Choices Under Uncertainty. Reading,Mass.: Addison-Wesley Publishing Comparty,1968.

7. Klarmah, Hi., "Application of Cost-BenefitAnalysis to the Health Serviies and tht Spe-cial Case of Technological Innovation. in-ternational Journal of Health Services 4:32352, Spring 1974. .

8. Bunker, J,P,, Barnes,.A.A. and Mosteller,- F.,Editors, Costs,, Risks and Benefits of Surgery.New York: Oxford, University Press, 1977.

9. Mushlin, A.1.; Appel, F.A. and Barr. D.M."Quality AssuTance in Primary Care: AStrategy Based' on Outcome Assessment."

Journal of Community Health, in press.

10. Jacobs, C.M. Christoffel, T.H. and laixon,Measurit' ig the Quality of Patient Care: The

Rationale for Outcome Audit. Canibridge,Mass.": Ballinger Publishing Company, 1976.

11.. L.L., "Quality Control and the Medi-cal Record." Archives of Internal Medicine127:101-105, January 1971.

a

12. Williamson, J.W., "Evaluating Quality of Pa- 20tient Care: A Strategy Relating Outcome andProcess Assessment." The Journal of the Ameri-can Medical Association 218:564.569, October25, 1971.

13. Brook, R.H.; Davies-Avery, .; Greenfield,S.; Harris, L.J.; Lejah, T.; Solomon, N.E. andWare, J.E. Jr., "Assessing the Quality ofMedical Care Using Outcome Measures: AnOverview of the Method." Medical Care Sup-plement to Vol. 15, No 9, September 1977.165 pp

14. Fanshel, S. and Bush, J.W., "A Health StatusIndex and kto Application to Health ServicesOutcome." Operation.; Research 18:1021-1060,November-December, 1970.

$15. Hirsch, E.O., "Utilization Review as' a Meansof Continuing Educajion." Medical Care12:358-$62, April 1974.

16. McNeil, -B.J.; Varady, P.D.; Burrows, B.A.and Adelstein, S.J., -"Cost-Effectiveness Cal-

.culations in the Diagnosis and Treatment ofHypertensive Renovascular Disease." NewEngland Journal of Medicine 293:216-221, July31, 1975. s

17. McNeil, B.J.; Hesse!, S.J.; Branch, W.T.;Bjork, L. and Adelstein, S.J., "Measures ofClinical Efficacy. III. The Value of the LungScan in the Evaluation of Young Patierits

pith .Pleuritic ess Pain." Journal of NuclearMedicine 17 :169-160, March 1976.

18. Neutra, R., "Indications for the SurgicalTreatment of Suspected' Acute Appendicitis:A Cost-Effectiveness Approach." In Bunker,J.P.; :Barnes, B.A. and Mosteller, F., Editors,Casts, Risks and Benefits of Surgery. New York:Oxford University Press, 1977:

19. ;Schwartz, 'W.A., Corry, G.A., Kassirer, J.P./and Essig, A., "Decision Analysis and Clinical

Judgment." The American Journal of Medicine55:459-472 (October 1973).

20. McNeil, B.J. and Adelstein, SJ., "The .Value37 of Case Finding in Hypertensive Renovascu-

lar Disease." New England journal of Medicine293121.2113 (July 31, 1975).

21. Abt, C.C., -The Issue of sal Costs inCostBenefit Analysis of Surgery." InBunker, J.P.; Barnes, B.A. and Mosteller, E..Editors, Cost). Risks and Benefits of Surgery.

New York: Oxfurrcl University Press, 1977.

22. Ginsberg, A.S.. Dicii4on Analysis in Clinical'Patient Management with an Application to thePleural-Effusion Syndrome. Santa Monica,Calif . Tire Rand Corporation,- July 1971,

23. Meyers, A.; Brand, D.A.; DoVe. H.G. andDolan, T.F. Jr.. A Technique for Analyzing

Clinical Data to Provide Patient Management

Guidelines: A Study of Meningitis in Children.

New Haven, Conn.: Center for the Study ofHealth Services, Yale University, January1975. ,

24. Sturdevant, R.A.L. and Stern, D., "Accuracyof Physicians' Predictions of Cholecystog-raphy Results." Medical Care 15:488-493,June 1977.

25. Piachaud, D. and Weddell, TM,. "The Eco-nomics of Treating Varicdnie Veins." Interna-tional Journal of Epidemiology. 1:287-294, Au-tumn, 1972.

26. Neuhauser, D. and Lewicki, A.M., "What DoWe Gain From the Sixth Stool Guaiac?" NewEngland Journal of Medicine 293:226-228, July

31, 1975.

27. Hardwick, D.E.; Vertinsky, P.; Barth, R.T.;MitChell, V.F.; Bernstein, M. and Vertinsky.I:, "Clinical Styles and Motivation: A Studyof Laboratory Test Use." Medical Care13:397-408, May 1975.

28. Martin, S.P.; Donaldson,C.D.; Peterson, O.L. and Coltinto Coronary Care During S

Effectiveness Study." A- Medicine 81:289-293 eptem

London,., "Inputsrs, A Cost

Internal, 1974.

29. Jackson, R.A. and mith, M. ., "RelationsBetween Price and Quality in ConicnunityPharmacy." Medical Care 12:32-39, Jiptuary

1974.

30. Schroeder, S.A.; Schliftman. A. and Piemme,T.E., "Variation Among Physicians in Use ofLaboratory Tests: Relation to Quality ofCare." Medical Care 12:709-713, August1974.

31. Rubenstein, L.; Mates, S. and Sidle,"Quality-of-Care Assessment by Process andOutcome Scoring: Use of Weighted Al-gOrithmic Assessment Criteria for Evaluationof Emergency Room Care of Women withSymptoms of Urinary Tract Infection." An- 45.

nals of Internal Medicine 86:617-625, May 3s1977.

32. Bloom, S.W. an I Wilson, R.N., "Patient.Practitioner Rel tioisships." 1u Freeman,11.F.: Levine, S. awl Reeder,. 1..G., Havtdbooli

of Medical Sociology, Englewood Cliffs, NJ.:Prentice-Hall list.. 91kimti Edition, 1972,

33. Lebow, 1.1... "Consumer Assessments of theQuality of Medical Care." Medical Caro12:328-337. April 1974.

34. Inui, T.S.; Yourtee. E.L. and Williamson,"Improved Outcomes in Hypertension

After Physician Tutorials: A ControlledTrial." Annals of Internal Medicine 84;846-851,June 1976.

35. Sackett, D.L.; Gibson, LS.; Taylor, D.W.;Haynes, R.B.; Hackett, B.C.; Roberts, R.S.and Johnson, A.L., "Randomised ClinicalTrial of Strategies for Improving MedicationCompliance in Primary Hypertension." TheLancet 2:1205-1207, May 31, 1975.

36. Smith, D.B. and Metzner, C.A.. "Differential,Perceptions of Health Care Quality in Pre-paid Group Practice." Medical Caro 8 :264-

275, July-August 1970.

37. Marrame S.D., "Patients' Evaluations ofNursing Performance." Nursing Research22:153-157, March-April 1973.

38. DeGeyndt, W., "Five Approaches to Assess-ing the. Quality of Care." Hospital Administra-

tion 15:21-42, Witer 1970.

39. Donabedian. A., "Some' Basic Issues inEvaluating the Quality of Health Care." Is-sues in Evaluation Research. Kansas City, Mo.:American Nurses' Association, 1976.

40. Shona. 9.M., "Continuity in Medical Caste:Conceptualization and MeasUrement. Medical

Care 14:377-391, May 1975.

41. Bass, R.D. and Windle, C., "Continuity ofCare: An Approach to Measurement." Ameri-

c Can Journal of Psychiatry 129:110-115, August1972.

42. Pugh, T.F. and McMahon, B., "Measurementof Discontinuity in. Psychiatric inpatientCare." Public Health Reports 82:533-538, June

1967. - r43. Mindlin, iR.L. and Densen, P.M., "Medical

Care of Vrban Infants: Continuity of Care."American Journal of Public ,Health 59;1294 -1301, August 1969.

44. Breslau, N. and Reeb, R.G., "Continuity ofCare in a University-Based. Practice." journalof Medical Education '50:965-969, October1975.

Breslau, N. kid Haug, M.R., "Service Deliv-ery Structure and Continuity of Care: A CateStudy of Pediatric Practice in Process of

r

Reorganisation." Aerosol te Health and SocialBehavior 17:339-352, December 1976.

40. Start leld, B.H.; Simborg, D.W..; Horn, Si),and Younee, S.A., "Continuity and Coordi-nation in Primary Care: Their Achievementand Utility." Medical Cary 14;625.636, July1978.

47. Bice. T.W. and Boxerman, Sit., "A Quan-titative

5:347-349, April 1977.sure of Continuity of Care."

Medical Ca

48. Short G.I. and. Nulling, LA. "APopulation-Based AssessmentLAd the Clui-tinuity of,' Ambulatory Care." Medical Gae15:455-464, June 1977.

49. Becker, M.H.; Drachman, R.11. and Kirscht,J.P., "A Field Experiment To Evaluate Vari-ous Outcomes of Continuity of PhysicianCare." American Journal of Public Health.64:1062-1070, November 1974,

50. Childs, .A.W. and Hunter. E.I)., "Non -Medical Factors Influencing Use of Diagnos-tic X-Rayby Physicians." Medical Care10:323-335, July-August, 1972.

51. Freeborn, D.K.; Baer, D.; Greenbelt. M.R.and Bailey, J.W., "Determinants of MedicalCare Utilization: Physicians' Use of Labora-tory services." American Journal of PublicHealth 62:846-853, June 1972.

52. Lyle,*C.B. Jr.; Citron. D.S.; Sugg, W.C. andWilliams, 0.D., "Cost of Medical Care 'in aPractice of Internal Medicine: A Study in aGroup of Seven Internists." Annals of internal.Medicine 81:1-6, July 1964.

55. Lyle, C.B. Jr.; Applegate, %N.B.; Citron. D.S.and Williams,e0.D.. "Practice Habits in aGroup of Eight ,Internists." Annals of InternalMedicine 84:394-601. May 1976.

54. Daniels, M. and Schroeder, S.A., "Variation,Among Physicians in Use of LaboratoryTests.' I. Iteration to Clinical Productivityand Ouicotnof Care." Medical Care 15:482-

.437;June 1977. ,

55. Schroeder, S.; Kenders. K.; per, . andPiemme, T.E., "Use of Laborat gists andPharmaceuticals: Variations. Among Physi-cians and Effect of Cost Audit on SubsequentUse." Journal of the American Medical Associa-tion 225 :969 -973, August 20; 1973.

56; Fattu, N.C., "Experimental Studies of Prob-lem. Solving." Journal of Medical Education39:212 -225, February 1964.

57. Kleinniuntz, B., The Processing of 'ClinicalInforthation by Man and Machine." InKfuntz, B.K., Edito , Formal Representation ofHessen Judgment N w York: John Wiley andSons, Inc., 3968.

58, Leaper, D.J.; P.W.; Stattiland. j.R.:"formats. J. and DeDomhal, E. .. "ClinicalDiagnostic Process: An Analysis." BritiaAMedical fining/ 3:569.574, Seinember, 15,1973,

MI. Hare. ILL. and Barnoon, S., Medical CareAppraisal and {lustily &SIMMS in the OfficePractice of Internal Medicine. San Francisco:American Society. of Internal Medicine, July1973.

60. Me Donald. C ..priv,x01.Based ComputerReminders, The Quality of Care and theNonPerfectability of Man." New England

Journal of Medicine 95:1351-1355, December9, 1976.

61. BroWn. C.R. and Uhl, S.M.. "MandatoryContinuing Education: Sense or Nonsense?"Journal of the American Medical Association213;1660-1668, September 7, 1970.

62. Kessner, D.M.; Kalk, C.E. and Singer. J.."Assessing Health QualityThe Case forTracers." New England journal of Medicine288:189-194, January 25, 1973.

63. Kessner, D.M.; Snow. C.K. and Singer. J..Assessment of Medical Care For Children.Washington Institute of Medicine.'1974.

64; Lyons. T.F. and Payne. B.C., "InterdiagnosisRelationships of Physician PerformanceMeasures." Medical. Care 12:369-374, Aril197.4.

65. Lyons, Ty. and Payne, B.C., "InterdiagltosisRelationships of Physician PerformanceMeasures in Hospitals." Medical Care15:475-481, June 1977.

66. Osborne, C.E., "Interdiagnosis RelationshipsPhysician Recording in Ambulatory Child

Health Care." Medical Care 15:465-474, June1977. r)

67. Payne, B.C.; Lyons, .T.F.; Dwarshius, L.;Kolton, M. and Mortis, W., The Quality ofMedical Care: Evaluation and Improvement.Chicago: Hospital Research and EducationalTrust, 1976,

68. Peterson, 0.L.; Andrews, L.P.; Spain, R.S.and Greenburg, B.G.. "An Analytical Studyof North Carolina General Practice, 1953 -1954." Journal of Medical Education Volume31, No. 12, December 1956, Part 2.

69. Morehead, M.A.; Donaldson, R. et al., A .

Study of the Quality of Hospital Cart Secured byA Sample of.' Teamster Fancily' Members in NewYork City. New York: Columbia UniversitySchool of Public Health and AdministrativeMedicine, 1964:

3g0. Brook, R.H. and Williams, K., "Evaluation of

31

32

the Ncw Mes144) Pert Review Ss aunt, 1071 to

1971" Afedual Carr Vol 14, No 12. Dr(ember 1976, Supplement 122 pp

71. Rher, S., "Fa41111% Deteimining the Qua Insof Physidan Pei lot MAIII r in Patient CateMedical l:nre 14:733-75. keptetillsel If176,

72, Scott, Wit.; Foliest, W 11 II and Blown.[VW.. "Ilospiial Slim lute mid Postoperative'Mortality and kfcllstilits In Shot tell. S 'N1

and BilIW II, SI , F4111111%, Organisational Research or ilospo441-s.. Clm_ago... Blue tacos, AA

%IX iiiii1111, 1976

73. lirrnk, R.11 'dill! VS'illiams. k N , "Qualits idHealth Care Inc the Disadvaritageil"10arnalol Comnuay llealth 1 13'.!-156, %%'itirei 1975

74. 1..vons, FF. and Pasiir. WC,. I hr Q41,1111% ill

Plivsiuji10%. Health/ "" 11" " AComparison Against OptiniAl Cum-it.' lotFiratmeni 4)1 hit Fldri Is and Votingri-Adidis in Commiiiiiis Hospitals." /owner/ 01

the Amernan Meduril .1%w, swum 227:92.5,928,

February 25, 1974.

75. Lyons, T.F and P.isite, It.(.. "Qt1.11ii of

Physician' (II lit c-Cate Pei hirman«.: DO lei

4..cut leo Fidel Is I can lot l'ourig,-, ;Willis?"journal 4,/ the .4rner:«in .1leeli«il .f .or sauna,229:16'21-1622, Spimbei 16. 197.1,

76. Griner. P.F. and Liptzin. B . "('sr 111 lite1,41)rivtmv in .1 I vat hiug Hospital: Impina-tinits Inr Kit-let:arc, 1%4614.464m, and Hos-

pital Crisis." .4eora/s 01 internal Medic:t.75: 157-I63Augnsi 1971,

77. Graham, J.B. and l'alourek, F. "WhereShould Cancer of die Cervix Be 1 leated?-Amencars Journal 01 (1b.tetru% and Gynecology

87:405-409, 094)be: 1963.

78. Schaffner, %N..: Fcrlei spiel, C.F.: hilton,M.L; Gilbert, D.G. and st,%(.11,;(m. ILA..

"Mdtertial Morialiis in !slit higair AnEpiderniolrigit Arials %is, 144,0- I 97 1." ,4mers-

ran Journal o/ Publu Health 67:M21-829. Sep-

tember 1977.

79. Hollinkshead, A.B. and Redlich, F.C., Social

Class acrd Mental lllnr.o. A l:orrirnuatty turly.

New York: John Wiles & Sims, Inc.. 1958.

80. Duff. R.S. and Hollingshead, A.B., Sickness

and Society. New York: Harper and Ross.-"--968.

81, utsteina D.D.; Berenberg, W.; Chalmers,1.C.'; :add, C. 3rd; Fishman, A.P. and

errin. E.B., "MeAtierints. the Quality ofedical Care: A Clinical Mettod." Nf IV Eng-

nd Journal of Medicine 294:582-588, March

1, 1976.

82. nabedian, A., Aspects of Medical Air Ad-

, stimistration: Specifying Requirentes for Health

MY

Cambildge, Mass Hai vaid UnivelittvNess. 1973

Paoli k, 1.. Rusk. I.W. and Chew M.M.,

I owai41 an Opriattnal DrIllittion ofI /m/ Health rind Social Hehavior14,6 23, Maids 1973.

8 Parini., D 1.. Hush. J.W. and (lien, M.M.,"Methods lot Mrassiiing levels ill Wrll-hnttig a Health Stains Index." NerjlthSerial's Ile,raih 8:228.245, Fall 1973.

4.144114..e, 1.61114k, WM. and Sackett,I) L. "A' lttilits Maximization Model furLvalualion of Health Cale Programs." HealthNero, el liesraidi 7: 118-133, Summer 1972.

Gilsnit, B.S.; J.S.. Brixiier M: ct al," I lit Sit klicv Impact 1111411c: Developinepi

4)1 Onteortie Measure of Ilealth Care."American journal of Puhlor Health 65:1304-13111, December 1975.

87, pollard. WE.; Bobbin. R.A.; Bergner, M.;Matins. add Gilson, B.S., "The SicknessImpact Prnfile:Reliabilily of a Ilealiti SiatusM canto c." Medico/ Care 14:146.155, February1976.

ttf4-

MM.

99.

.91

92

Brig, RI.., Editor. Health Status indexes.Chit-agb: Hospital Reseirch and Educational1 nisi. 1973.

Elitism), J., Editor. "Socionicilical Health In-dicators." International Journal o/ Health Serv-

ices 6:337.538, 197(1.

Several Authors, "Health Stains Indexes-Work in Progress." Health Services Research11:332-528, Winter, 1976.

Breslow. I.., "A Quantitative Approach to theWorld Wahl) Organization Delitiitirm ofHealth: Physical, Menial and Social Well-beitig.'; international Journal of Epidemiology1:347-355, Winter 1972.

Coditian, F..A.., A Study in Hospital EfficiencyAs Demonstrated In the Case Report of the FirstFive Years of a Private Hospital. Boston:Thomas Todd Co., Circa 1916.

93. Hulka, B.S.; Zyzinski, S.J.; Cassel, J.C. and"Fhompsyn,- Si., "Scale for the Measurementof Attitudes toward Physicians and PrimaryMedical Care.- Medical Care 8:4449-435.September-October 1970.

94. Hulka, B.S.; Zyzanski, S.J.; Cassel, J.C. and. Thompson, "Satisfaction with MedicalCare in a Low Income Population. Journal of

'Chronic Disease 24:661-673. November 1971.

95. Zyzanski,- S.J.; Hulka, B.S. and Cassel, IC.:"Scale for the Measurement of 'Satisfaction'with Medical Care: Modifications in Content,Format and ScoPing." Medical Care 12:611- .

0 620. July 1974.

96. .Sehmidt, E.C.; Schall, D.W. and Morrison,C.C., "Computerized Problem-OrientedMedical -Record For Ambulatory Practice."Medical Care 12:310427, April 1974.

7. Grimni, R.H.; Shimoni; K; Harlan, W.R. and'Estes, E.A. Jr., "Evaluation of Patient-CareProtocol Use by Various Providers." NewEngland Journal of Medicine 292:507-511,March 6, 1975.

*98. Lembcke, P.A.., "M ical Auditing by Scien-tific Methods." Jou 1 of the American MedicalAssociation 162:64 -655, October13, 1956.

99._Wiener, S. and Nathanson, M., "PhysicalExamination: Frequently Observed Errors."Journal of the American Medical Asiociaton236:852-855," August 16, 1976.

100. TUrnet, E.V.; Helper, M.M. and Kriska,S.D., "PrediCtors of Clinical Performance."

Journal of Methcal Educatio'n 47:959-965, De-cember 1972.

a-101. Zuckerman, A.E.; Starfield, B.; Hochreiter,

C. and Kovasznay, B., "Validating the Con-tent of Pediatric Outpatient Medical Recordsbi Means of Tape-ReCording -Doctor-PatientEnconniers." Pediatrics 56:407-411, Sep-tember 1975.

102. Steward, M.A. -and Buck, S.W., "Physicians'Knowledge of and Response to Patients'Koblems." Medical Care 15:578, July 1977.

103. Tufo, H.M..; Eddy, .M.; Van Buren, H.C.;Bouchard,-.R.W.; Twitchell,.K. and Bedard,L., "Audit in. a Practice Group." Pages 29-41In Walker, H.K.; Hurst, J.W. and Woody,M.F., Editors, Applying the Problem-OrientedSystem. NewYork: Medcom Press, 1973.

104. SimbOrg; D.W.; Stanfield, B.H.; Horn, S.D.and YOurtee, S.A., "Information Factors Af-fecting Problem Follow-Up in AmbulatoryCare." Medical. Care 14:848-856, OctOber1976.

103. Grover, M. and Greenberg, T.SQuality ofCare Given to First Time Birth Control Pa-tients at a Free Clinic." American Journal ofPublic Health ,66:9867987, October 1976.

106. Wassertheil7Smollei, g.; Bell, B. and Blaufox,M.D., "Computer-Aided Management OfHypertensive Patients." Medical Care13:1044-1054, DeceMber, 1975.

107.. Barnett, G.0,; Winickoff, R.; Dorsey, J.L.and Morgan, M.; "The. Role of Feedback inQuality Assurance-An Application ofComputer-Based Ambulatory. Medical In-formation 'System." In Assessing PhysicianPerformance in Ambulatory Care. 'San Francisco:AmericanSociety of internal Medicine, 1976.

108. ROstnberg, E.W., 4!What,Kind of Criteria?"

Medical Care 13:966-975, November 1975.

109. Greenfield, S.; Lewis, C.E.; Kaplan, S.H. andDavidson, M.B., 'Teti' Review by CriteriaMapping: Criteria for Diabetes Mellitus, TheUse of Decision-Making in. Chart Audit."Annals of Internal Methane 83:761-770, De7,.cemller 1975.

110. Richardson; F.McD., "Methodological De-velopment of a System of Medical Audit."Medical Care 10:451-462, November-Deceniber 1972.

1 ft.:tIoekelman, R.A. and Peters, .E.N., "ASupervision Index to-Meatire-Stand="

ards of Child ,Care." Health Services Reports87:537-544, June-July 1972:

112. Lyons, T.F. and Payne, B.C.., "The Use ofItem Importance Weights in Assessing Physi-:clan 'Performance with PredeterminedCriteria Indices.:Medical Care 13:432-439,May 1975.

.113. Hopkins, C.E.; Hetherington, R.W. and Par-sons, E.M., "Quality of Medical Care: AFactor Analysis Approach Using MedicalRecords." Health Services Research 10:199-208,Summer 1975..

114. Novick, N.R.; Dickinson, .1(; Asnes, R.; Lan,M. S-P. and Lowenstein, R., "Assessment ofAmbulatory Care: Application of TracerMethodology." Medical Care 14:1.12, January1976.

113. Donabedian, A. and Wyszewianski, L., "TheNumerology of utilization Control Revisited:When to Recertify." inquiry 14:96-102,March 1977.

116. Institute of Medicine, Assessing Quality in.Health Care: An Evaluation. Washington,D.C.: National Academy. of Scienes,November 1976;

117. Averill, R.F. and McMahon, L.F., "A CostBenefit Analysis of Continued Stay Certifi-cation." Medical Care 15:158-173, February1977.

118. Wolfe, H., "A Computeriied Screening De-vice for Selecting Cases for Utilization Re-view." Medical Care 5 :44 -51; JanuaryFebruary 1967.'

119. Riedel, D.C.; Fett r, R.B.; Mills, R.E. andPallett, PJ., Basic tilization ReView Program(BURP). New Hay , Cohn.; Health Services'research Progra Yale Unidltrity, 1973.

120. Rubin, L., Comprehensive Quality AssuranceSystem: The Kaiser-Permanente Approach.Alexandria, Va.: American Group PracticeAssociation, 1975.

121. Glass, R.I.; Mulvhill, M.N.; Smith, H. Jr.;Peto, R.; Bucheister, D. and Stoll, BJ., "The

41

4 Scqre: An index for Predicting A Patients'Non-Medical Hbspital Days." American Jo Ur-

ofPublic Health 67:751-755, August 1977.

122. Brauer, L.D., "Selection of Cases for Indi-vidual' Review." In Riedel, D.C.; Tischler,G.L. and Myers J.K., Patient Care Evaluationin Mental Health Programs. Cambridge, Mass.:Ballinger PublislAing Company, 1974.

123. Donabedian,, A., A Guide to Medical Care Ad-.ministration, Volume II: Medical CareAppraisal-QUality and Utilization. New York 135

(now Washington): American Public Health_:___Asso'ciation,1,969,1.176 pp. (plus Annotated

Sete ted Bibliography by A.J.

134.

34 124. )ook, R.H. and Appel, F.A.: "Quality-of-Care Assessment: Choosing' a Method forPeer Review." New. England Journal ofMedicine 288:1323-1329, June 21.,,1973.

125. Brook, R.H., Quality of 'Care Assessment: AComparison of Five Methods of Peer Review.Washington, D.C.: U.S. Department ofHealth, Education and Welfare, Bureau ofHealth Services Research and Evaluation, 13

July 1973126. New York Academy of Medicine,)Corrunittee

on Public. Health Relations, Maternal Mortalityin New York City: A Study of All PuerperalDeaths, 1930-32. The . Commonwealth Fund,New York: Oxford University Press, 1933.

127. New York Academy of Medicine, Committeeon Public Health Relations, Petinatal Mortalityin New York. City: Responsible Factors. Cam-bridge, Mass. Harvard University Press, forthe Commonwealth Fund, 1955,.

128. Oakley, J.R.; McWeeny, P.M.; Hayes-Allen,M. and Emery, J.L., "Possibly AvoidableDeathS in Hospital in the Age-Group OneWeek to Two Years." The Lancet 1:770-772,April 10, 1976.

129. Gonnella, J.S. and Goran,1 M.J., "Quality ofPatient Care-A Measurement of Changei,

The Staging Concept."- Medical Care 13:467-,473, June 1975.

130. Gonnella, J.S.; Louis, D.Z. and McCord, JJ-."The Staging Concept-An Approach to theAssessment of Odtcome of AmbulatoryCare." Medical Care 14:13-21, January 1976.

.131. Brook, R.H. and Stevenson, R.L., "Effective-ness of Patient Care in an Emergency Room."New England Journal ofMedicine 283:904-907,October 22, 1970.

132. Starfield, B. and Scheff, D., "EffeCtiveness ofPediatric Care: The Relationship BetweenProcess and Outcome." Pediatrics 49:547-552,April 1972.

133. McCarthy, E.G. and Widmer, G.W., "Effects

of Screening by Consultants on 'Recom-mended Elective Surgical Procedures." NewEngland Journal of Medicine 291:1331-1335,December 19, 1974.

McCarthy, E.G., "Mandatory and 'VoluntarySecond Opinion Programs in the GreaterNew York Area with National Implications."The Dorothy Eisenbeip Lecture, Departmentof Surgery, Harvard Medical School, June12, 1976.

. Freeborn, D.K. and Greenlick, M.R., "Evalu-ation of the Performance of AmbulatoryCafe Systems: Research Requirements and

March-April 1973.

136. Christoffei, T. and LoWenthal, H., "Evaluat-ing the Quality of Ambulatory Health Care:A Review of Emerging Methodologies." Med-ical Care 15:877-897, November 1977.

137. American Society of Internal Medicine, As-sessing Physician Performance in AmbulatoryCare. San Francisco: The Society, .1976.

8. American Nurses' Association-, Issues inEvaluation Research. Kansas City, Mo.: TheAssociation; 1976.

139. Several Authors, "Symposium on the Qualityof Pharmaceutical Services." Drugs in HealthCare 2:3-66, Winter 1975. .

140. Knapp, D.A. and Smith, - M.C., EvaluatingPharmacists and Their Activities. Washington,D.C.: American Society of Hospital Phafina7cists, Researth and Education FoundatiOn,1973.

141. Several papers in the Journal of the AmericanDental Association 89: 842-871, October 1974.

142: Riedel, D.C.; Tischler, G.L. and Myers, J.K.,Editors,' Patient Care EvalUiltion in MentalHealth Programi. Cambridge,' Mass,: BallingerPublishing Company, 1974.

.

143. Ehrlich, J.; Morehead, M.A. and, Trussell,R.E., The Quantity,. Quality and. Costs of Mediial,and Hospital Care Secured by a Sainple of Teams-.ter Families in the New York Area New York:

Columbia Unive ity, School of Public Health -

and Administra ve*Meclicine, 1962.

144. Shenkin, B.N. and Warner, D.C., "Giving the.Patient His Medical Record: .A Proposal toImprove the System." New England Journal ofMedicine 289:688-692, September 27, 1973.

145. Stevens, P.D.; Stagg, R. and Mackay; I.R.,"What Happens When Hospitalized PatientsSee Their OWn Records." Annals of InternalMedicine 86:474-477, April 1977.

146. Bouchard, R.E.; Tufo, 1.1.M,;- Van Baren,H.C.; Eddy, W.M.; Twitchell; J.C. and Be-dard, L., "The Patient and His Problem-

147;

Oriented Recoilr." In Walker, H.K.; HutstsJ.W. and Woody, M.F., Editors, Applying theProblem-Oriented System. New York: MeFon)Press, 1973.

Burger, C.S., "Audit in a Problern-OrientedPractice." In Walker, H.K.; :Hurstd.W.and Woody, MS., Editors,Apply'ing the Prob-lems-Oriented 'System. New Vcirk: MedcomPress, 1973.

148. Goldblatt, P.I3.; Henisz, J.E. and Tischler,G.L, "Utilization *view Within an Institu-tiOnal Context." in Riedel, D.E.; Tischler,G.L. and Myers, J.K., Editors, Patient CareEtiglua '''in-71te're:tai Health 'Programs.bridge, Mass.: Bcallinger Publishing Com-pany,,1974,.pany, 1974: 292 pp.

149. Donabedian, A., "Promising QualityThrough Evaluating the ProCess of PatientCare," Medical Care 6:1811202, May-June1968.

Regulation, June, 1975. DHEW PublicationNo.. (HRA) 77-621, Waihi gton, D.C.

151. Morehead, M.A., "P.S.R. Problems andPossibilities." Man and Medicine 1:11'3-132,Winter 1976.

152. Bernd, L.E., "PSRO Quality Control? OrGiinmickry?" Medical Care 12:012-1018, De-cember 1974

153. Brian, E.W., "Government control of Hospi-.tal Utilization: A California Expetience."NewEngland JoUrnal of Medicine 280340-1344,June 22, 1972.

..... 454-:---Glendenning; M:K.T Wolfe; -H.; Sh4m an, L.J.. and Hither, G.A., "The Effect of a Target. 35

Date Based UtilYzation Review Program onLength of Stay." MediCal Care. 14:751-764,Sep r 1976.

155. Lave, J.R. nd Leinhardt, S., lekn Evaluationof a Hosp I Stay' Regulatory. Mechanism."American Journal of Public Health 66:959-967,'October 1976.

156. Wyszewianski, L., work in progress, Depart-inent of Medical Care Organization,, TheUniversity of Michigan, Ann. A0or.

150. Freidson, E., "Speculations on the SocialPsychology of Local PSRO Operations." InProceedings: Conference on Professional Self-

.

43

National Center for Health Services Research pub-lications of interest to the health community areavailable on request to NCHSR, Office of Scien-tific and Technical Inforination, 3700 East-WestHighway, Row 7-44, Hyattsville, MD 20782 (tele-phone: 301/436-8970). Mail requests will be facili-tated by enclosure ofaielf-addressed self-adhesivemiffing label. These. publicaficms also are availablefor sale.'through the National Technical Informa-Lion Service (NTIS), SPringfield; VA 22161 (tele-phone: 703/557,4650). PB and HRP 'nurnbers inparentheses are NTIS order numbers.. Publication,Swhich are out of stock. in NCHSR are indicated' asavailable only from NTIS. Prices may be obtainedfrom the NTIS order desk on iliquest.

Research Dig

The Research est SO provides overviews ofsignificant researc pported" by ..NCHSR. The

.A'serieideie 'lies either ongoing or completed prof-

ects direc toward. high priority health services

probleM . Issues are prepared by the principal investigators perforMing the research, in collabora-tion with NCHSR 'staff. Digekts are intended foran interdisciplinary audienci of health servicesplanhers, administrators, legislators, and -otherswho make decisions on research' applications.

, .

Evaluadon of a Medical Information Systemin a Community Hospital (PB 264 353) .

CoMputer-Stored Ambulatory Record(COSTAR) (PB 268-342)

Prcram Analysiti of plipiclay Extendei Al-gorithm, Projec (P.B...264"e10, availableNTIS only) .

Changes in the Costs of T-reatment of.Selected. Illnesses, 1951-1964-1971 (HRP001'4598)

Impact of State Certificate-of-Need Laws onHealth Care Costs and Utilization (PB 264

352)"

(HRA) 78-3144

(HRA) 78-3145

(HRA)77 -3180

(HRA) 77-3161

v

(HRA) 77-3163

(HRA) 77-3164 An Evaluation of Physician Assistants In

'Diagnostic Radiology (PB 286 507, avail-

able NTIS only)c,..

(HRA) 77-3166

(HRA) 774171

(HRA) 77-3.173

(HRA) 77-3177

Foreign Medical S aduates: A ComparativeStudy of State nsure Policies (PB 265233) i'.Analysis of Physician Pt,rice and Output De-cisions (PB 273 312)

Nurse Prietponer and Physician AssistantTraining and Deployment (PB 271.000)available NTIS only

Automation, of the Problem-Oriented Medi-cal Record (PB 286 88 1 )

Research Summaries

The Research Sum ty Series provides rapid aeCessto significant resu s 'ol NCHSR-supported re-.

'arch projects. T e :series presentx.ekec the

summaries prepare by the investigators at Ghe

completion of the project. Screlific findihigleighedin a morecontise form than in threport. The Research Summary Seri". iiintend forhealth services administraiors, planners, and 'otherresearch. users' who require recent findings rel-'-eyant to immediate programs in health services.

(HRA) 7:3182

(HRA) 77-3183

(HRA) 77-3176ar,

(HRA) 78-3193

Recent Stiitilqs. , Nealt Services Re-search, Vol. 1. 974 t oughDecom-ber 1976) (PB 20 480

Recent Studies I,n Health ServiceS Re-search, Vol. II (CY 1976) (PB 279 198)

Citality 'of Medical Care 'AssessMent UetgOutcome Measures (P8-272 455)

Optimal Electrocardiography (PB 281' 558)

Policy Research,

.

The Policy Research Series desCribes findings from,the research program that have major signicc,ancefor policy issues of the moment. These papers areprepared by members of the staff of NCHSR or byindependent investigators. The series is intendedspecifically to inform those in the public gild pritvats sectors who must consider, design, and ini-

.. plemein policies affecting the delivery of ,health"service's.

4 4

:

(HRA) **74162 .Controlling thicost of Health Care (PB 288885)

Research Reports

'r The Research Report Series provides significant re-.search reportsln their entirety upon the comple-tion of the ,projeq.-geseareh Reports are yle-veloped by the prince al investigators who con-duCted the research, and are directed to selectedusers of-health services research as part of a con-tinuing' NCHSR effort to expedite the dissernina-don of new knowledge resulting frill its, projectsupport.

(HRA)'76-3143 Computer Based Pail nt Monitoring Sys-tems (PB 286 508) ,4

(HRA} 77-3152 How Lawyers Handle Medical MalpractiCeCases (HRP 0014313)

An Analysis of the Southern California Ar-bitration Project, January 1988 thtoughJune 1975 (HRP 0012466)

(HRA) 77-3159

(HRA) 77.3165

HRA) 77.3184

(HRA) 77-3188

: (IKA) 74189

(PHS) 78-3204

Statutory Provisions for "Binding ArbitrationOf Medical Malpractice Cases (PS 264 40ftavailable NTIS only) , ,

'1g60 afici 1970 Spanish Heritage Fopula-.

tion of the Southwest County (PB 280656)

,Pemonstration and valuation. of a TotalHospital Information System PB 271 079)

Drug Coverrifie under National Health In-surance: The Policy Options (PB 272 074)

.Experiments in interviewing, Techniques:Field. Experiments in Health Reporting (PB276 080) /

Research Management

The Research Management Series describes "pro-grammatic .rather than technical aspects of theNCHSR research effort. Information is presentedon rthe,'NCEISR goals, research objectives, apriorities; in addition, this series contaim listsgrants and contracts, and administrative informtiOn on funding. Publications in this series are in-

<tended to bring basic. information 6n NCHSR andits prograrns to research plamirs, administrators,and others vibo are involved with the allocation ofresearch resources.

(HRA) 76-3136

(HRA) 77-3158

(HRA) 77-3187

(HRA) 77.3179)

(HRA) 77-3194

HRA) 7S-

o*

The Program in Health Services Research'(Ravised 9178)

Summary of Grants and Contracts, ActiveJUne 30; 1976.

EmergenCy Medical Senitces Systems Re-search Projects .( Active as of June 30,1976) (PB 264 407, avallable.NTIS only)

Research on the Priority issues of the Na-,tional Center of Health Services Research,Grants and Contracts Active on June 30,1976 'Emergency Medical Services Systems 9e-search Projects, 1977 (PB 273 893).:

4.KCI4SELResearch..BibilogrePhy- (July--11978 to June 30, 1977) (PB 273 997)

.37

Research Proceedings

Research Proceedings Series extends the avail-ability of new research announced at key confer=ences; symposia and seminars sponsored or sup-ported by NCHSR. In additiOn to papers pre.sented, publications -in this series includes discus-sions andoresponses whenever possible. The series

intended to help meet the information needs of.health services providers and others who requiredirer ccess to concepts and ideas evolving fromthe ex ange of research results.

(HRA) 76-3138

(HRA) 78-3150

(HRA) 77-3154,

(HRA) 77-3181

(HRA) 77-3186

(PHS) 78-319t

Women and Their eith: Research impli-cations- for a New E (VIP 264-359, avail-able NTIS.only)

interniountain Med cal Malpractice (PB 288344', available NTIS only) .

Advances In Health Survey 'RetearchMethods (PB 282 230

NCHSR Research Conference Report .onConsumer Self -Ctrs. in Health (PB 273 811)

intern'ational Cbnference on Drug andPharmaceutical Services Reimbursement(PB 271 388)

EmergenCy- Medical. Services: ResearchMethodology (PB 279 098)

v.

Probram Solicitation.

(HRA) 77-3196

(HRA) 77-320-0

(PHS) 78-3206

4.5

Conference Grant Information

Grants for Dissertation Research Support

Grants for Cost attainment Research. forHealth ,Planning

BIBLIOGRAPHIC DATA I. Report No. ; ,3, Recipient's Accession No-

SHEET NCHSR 78-145-

4. Title and Subtitle ..N..., 5. Report DateNEEDED RESEARCH IN THE ASSESSMENT AND MONITOR GIOF THE July 1978

. .

QUALf pp. MEDICAL CARE; NCHSR Research Report Sgrias.

,..,', t7. Auth 6r(s)

8. Performing OrganizStion Relit.Avedis Donabedian . Igo:,9. Performing Organization Name and Address

10. Project/Task/Work Unit No.The University of Michigan.School of Public Health

. 11. Contract/Grant No.Department of MedicalCare Organization' ....

-Ann-Arbor, MI. 48104 AC 313/764-5433 .HS 02081

12. Sponsoring Organization.Narne ape Address 13. Type of Rietrolotrte:iPmeriltDHEW, PHS, OASH, Hational Center for Health Services Research

-4- /30/793700 East -West Highway, Rooi 7-44 (STIB) 5/1/76-%

. HyOtsville, MD 20782AC'301/436=8970 14.

15; SupPlementary Notes '.

,DHEW,Pub. No. (PHS) 78-3219. The final report will not be. completed until approx. June1979. This is an interim reiOrt.-

16. Abstracts . .

/ The purpose of this report is to review, evaluate critically, and synthesize the-liter-ature on quality assessment and assurance, including the appropriateness Of use ofservice, in order to arrive at a ,cogent, documented, and authoritative assessment ofthe state -of- the -art. In addition )to addressing quality assessment as a research tooland quality assurance aslAn acflAistritive tool, an attempt is made to provide an up--derstanding of the epidemiology Of quality as a prerequisite to the.desigh of medicalcare prograMs and - systems. Major components af quality which are discossed include:(1) definitions, (2) quality Assessment and pr gram evaluation, (3) relationship ofquality and quantityl (4) relationship of qua. ty a cost,'(5) strategies of care,(6) structure, process, and outcome, (7) monitorin versus research, and (8) the uses.Of outcomes. Recommendations f further,research in the assessment and 'monitoring of

1 the quality of medical'. care are, presented.S

.

1Tribey-VaretrarrtImbarvinvetnn4ite+yriv7-49er./4..;...tiprora- '.1

.

. .HCHSRpublication of research findings does not necessarily represent approval or' '.

'official endorsement by,the National Center for-Health Services Research or theU.S. Department ol'Health, Education, and Welfare.

.

.

. . , .

Diane N. Funk: NCHSR P.O.; V1301/436-8941:1

4.. .

17b. IdentifierS/Open-Ended Terms.

.

Health aervices research ..

Quality of medical careQuality assessment

,

t

Quality assuranceMonitoring the,quality,of care

.

, .

.

.. .

.

17c. COSATI Field 'Group 018. Availability Statement 19. Security Class /This 21. No. of PagesReleasable to the public. Available from National Report)

Est.: 50.Technical Information Service, Sp ingfield, VA g ecurity C ass ( is 22. PriceAC 703/557-4650 22161 PageUNCLASSIFIED

44014M 447>S-35 MEV- 10,73> ENDORSED BY ANSI A D UNESCO. THIS FORMMAY BE REPRODUCEDi

6,

uscomm-4C 5265-P74