Methodological advances in the use of recidivism rates to assess mental health treatment programs

Methodological Advances in the Use of Recidivism Rates to Assess Mental Health Treatment Programs

Thomas L. Hafemeister, J.D., Ph.D. Steven M. Banks, Ph.D.

Abstract This study of recidivism rates within a state psychiatric system identifies and controls for a

number of flaws in other recidivism studies and provides one of the few direct comparisons between different mental health programs using recidivism rates as a dependent variable. It is not an attempt to predict the likelihood of an individual patient's recidivism or to improve the treatment provided the individual patient. Instead, it is an endeavor to address the issue of using recidivism rates to assess a treatment program's effectiveness and to examine such treatment at the macro- or systemwide level.

Introduction

Renewed Interest in Inpatient Care

The use of a hospital setting to care for individuals with mental illness is once again becoming a matter of heightened concern in this country. Historically, the interest of the general public in this care has been minimal except following relatively brief expos6s alleging grossly inadequate care within such facilities.1 Recently, however, public attention has been more consistently trained upon the inpatient care provided by facilities for individuals with mental illness. This increased attention can be traced to a number of factors.

First, there is greater recognition of the pervasiveness of mental illness and the increasing use of psychiatric inpatient services in this country. 2'3 Recent proposals regarding national health care reform have focused particular attention on the appropriate level of psychiatric inpatient services. At the same time, many states recently have instituted or considered changes in their civil commitment laws to broaden the range of individuals who may be involuntarily institutionalized, with data suggesting that such changes may result in major increases at least in state psychiatric hospital admissions (e.g., revised laws in Washington may have contributed to a 91% increase during the following year). 2'46

Second, the cost of providing inpatient care to individuals with mental illness has risen to unparalleled levels, with estimates that 70% of the total expenditures for mental health care in the United States goes for inpatien t care and that these costs will continue to accelerate well in excess of the inflation rate. 2'3 Meanwhile, the difference between the cost of residential and hospital treatment has been set at $500 per day. 7 Fiscal crises in many states and within the federal government

Address correspondence to Thomas L. Hafemeister, J.D., Ph.D., Senior Research Associate, Institute on Mental Disability and the Law, National Center for State Courts, 300 Newport Ave., Williamsburg, VA 23187-8798.

Steven M. Banks, Ph.D., is the director of statistics at the Bureau of Planning, Assistance, and Coordination, Office of Mental Health, State of New York.

190 The Journal of Mental Health Administration 23:2 Spring 1996

have accentuated interest in the cost of these services, means of minimizing or limiting their cost, and the possibility of using less costly treatment alternatives, s-H

Third, since the mid-1950s there has been continuing controversy over whether it is more appropriate to use inpatient hospitalization or community placements to care for individuals with mental illness. Although commentators today generally agree that the latter is the preferred site for such care, most also acknowledge that the former needs to be available for certain individuals during certain periods of time. The current debate tends to center on whether inpatient care is provided often enough; whether patients leave hospitals (usually state psychiatric facilities) before they are adequately prepared to survive in the community, leading to relapses necessitating rehospitalization (the so-called revolving door phenomena); and whether, as a result, large numbers of individuals with mental illness are left unattended or underattended in the community. 5'12~6 Furthermore, although the decrease in census in state psychiatric facilities most often associated with deinstitutionalization has slowed somewhat in recent years, it has nonetheless continued and, indeed, received new impetus from proponents of an intensive case management system.17:~ Finally, the plight of the homeless, particularly that portion of the homeless population that suffer from mental illness, has refocused attention on this issue. ~2~2"24

Fourth, the care that the hospitalized mentally ill receive has become more politicized and publicized, with its various facets increasingly subjected to scrutiny and public debate. 2'25 Histori- cally, relatives and friends of the mentally ill have been poorly organized and an ineffective lobby for drawing attention to such issues. Little oversight was given to the care provided within psychiatric hospitals. However, the number and activity of advocacy groups, both official and unofficial, has recently expanded, as well as the number of bodies overseeing the care provided within psychiatric facilities. 26'27 Among the latter are the Joint Commission on Accreditation of Health Organizations (JCAHO), the federal government's Health Care Financing Administration (HCFA) and Department of Justice, third party payors, independent state agencies, and local protection and advocacy groups funded by the federal government pursuant to Public Law 99-319. In addition, the traditional checks on psychiatric hospitai care (e.g., district attorneys initiating grand jury investigations and malpractice and other tort suits aimed at specific cases of alleged inappro- priate care) appear to be showing renewed vigor.

These factors have led to a great increase in the attention given the care of the mentally ill in general, and the care provided within inpatient settings in particular. Despite efforts to lessen reliance on hospitals as psychiatric care providers, it has been asserted that between 70 and 75% of the total direct mental health expenditures in the United States are for psychiatric hospitalizations. 3 Indeed, as noted in a recent congressional report, "Mental health professionals, State and local officials, and others are discussing the possible need for a reinstitutionalization movement" (p. 4). 2 As a result, attention continues to focus on the efficacy of the careprovided within hospital settings. In particular, this attention has focused on state psychiatric facilities.

Assessing the Efficacy of Inpatient Care The criticisms and concerns raised over the system of care provided to individuals hospitalized

for treatment of their mental illness have two primary loci. First, what is the quality of the care provided within psychiatric facilities? Second, can steps be taken during the course ofhospitaiization to maximize the ability of the mentally ill individual who has received psychiatric inpatient care to survive and remain in the community?

Although there has been considerable controversy over how best to assess these two aspects of inpatient care, a single method, namely, the recidivism rate of discharged patients, has tended to be used to measure both. There are numerous variations of this measure, but their underlying assumption is that the longer an individual lives in the community without readmission to an inpatient facility, the more successful or efficacious the preceding inpatient treatment.

Methodological Advances HAFEMEISTER, BANKS 191

Numerous reasons are given for assessing and relying on the recidivism rates for discharged psychiatric patients. 3~s3~ However, at the same time, studies relying on recidivism rates to assess treatment effectiveness have been plagued with methodological flaws, including (1) failure to accurately track the treatment history of the individual, (2) insufficient time span and data points, (3) lack of replication across programs and settings, (4) failure to randomly assign subjects to treatment and control groups, (5) failure to specify the relevant program/treatment, (6) failure to control for intervening variables, and (7) insufficient sample size to control for individual/patient characteristics. TM

With this myriad of potential confounds, it is perhaps not surprising that most recidivism studies fall to find variables that adequately predict hospital readmissions and, more specifically, have found that the type of inpatient treatment has little impact on recidivism rates. 3'~'29z: Two of the major reviews of studies examining readmission rates in general concluded that the only variable consistently related to rehospitalization was the number of prior hospitalizations. 3'3~ The perhaps most comprehensive study that attempted to identify individual predictor variables, using a stepwise multiple-regression analysis with a large number of variables, found that none of the predictors was strongly related to recidivism. 33 The authors concluded, "The ambiguity of these results resembles inconsistencies seen in findings from prior studies, and suggests that recidivism is a complex phenomenon which can neither be predicted nor explained through any single factor but only by examining the interactions of a variety of factors" (p. 32). 33

Reported Recidivism Rates Certainly, there is considerable importance in understanding what influences recidivismrates. In

addition to its potential impact on the ongoing debate over the appropriateness of inpatient versus community care discussed earlier, readmissions, by all accounts, constitute a large portion of psychiatric admissions, with an accompanying impact on hospital census and the nature of the treatment plan provided to such individuals. In addition, the need for a psychiatric readmission represents considerable potential for disruption, trauma, and danger for the discharged patient and those individuals who interact with the discharged patient. Although recent data are generally lacking, it has been reported that in 1950, only 25% of statehospital admissions had prior admissions to any type of psychiatric facility but that this rose to 48% in 1962, 64% in 1972, and 69% in 1975. TM

Furthermore, although commentators have been unable to identify confidently variables that predict recidivism, they have almost universally included it as one of the key outcome measures in assessing a discharged patient's successful reintegration into t h e community. 3'7'1723'2s3~

In their often-cited review of recidivism studies, Anthony, Buell, Sharratt, et al.:s concluded that the recidivism base rate (i.e., not taking into account any specialized treatment plan, aftercare follow-up, or particular group of patients) for hospitalized psychiatric patients was 30-40% six months after discharge, 40-50% one year after discharge, and 65-75% three to five years after discharge. Minkoff, 3~ reviewing seven articles published between 1967 and 1977, concluded that recidivism rates tended to be somewhat lower than reported by Anthony, Buell, Sharratt, et al. :s

Various lengths of time, ranging from six months to five years, have been used to assess the fimctloning of individuals discharged from psychiatric hospitals. Although some differences have been found between one-year and two-year follow-up periods, 34 differences were not found at two-year and five-year f o l l o w - u p s . 36 Generally, however, few researchers have attempted to assess the progression of recidivism across time for identified individuals, perhaps because of the magni- tude of such studies.

Differences Between Hospitals Generally, there appears to be an assumption that recidivism rates vary little from hospital to

hospital when differences in patient composition, treatment modalities, and community programs are taken into account. Similarly, it appears to be assumed that where these variables remain constant,


within-hospital recidivism rates will remain constant over time. With the following exception, psychiatric recidivism studies have not focused on between- or within-hospital differences.

Weinstein, Dipasquale, and Winsor 37 did compare what they termed the reentry rate for the 26 state hospitals operating during fiscal year 1971 in New York State. They found a wide difference in reentry rates after six months for the patients discharged from these hospitals, ranging from 19:5 to 37.2 %. They did not specify which hospitals represented the extremes of this range, or what factors may have led to these differences. Interestingly, they found that there was no substantial difference between the reentry rate for those state hospitals located in New York City (27.0%) and those located in the remainder of the state (28.6%), suggesting that between-hospital differences were not a result of regional or urban versus rural differences. In addition to not providing insight into the causes of between-hospital differences, the study also provides only the single temporal snapshot of the reentry rate at the six-month mark. It is possible that the between-hospital differences change or eviscerate for different time periods after discharge and at different points in time.

New York State In many ways, the status of the mentally ill in New York has reflected that of the natiOn as a

whole. New York was at the vanguard of the deinstitutionalization movement, with steps already initiated in the late 1960s to carry it out. A drop of 64% in the resident population of the state mental institutions in New York was mandated between 1968 and 1973, and the discharge and readmission of psychiatric inpatients in New York was frequently targeted as an example of the revolving door phenomenon in mental health care. 3~ New York has shown (1) decreases in its total state hospital inpatient population (as well as in the rate at which this has slowed in recent years) 16 and median lengths of stay for patients in these facilities (comparing data of Minkoff 3~ and Weinstein, Di- pasquale, Winsor 37) similar to the nation as a whole; (2) similar increases in its expenditures to support this census; 3s and (3) roughly the same number of psychiatric inpatient additions (admissions, readmissions, transfers, and returns from leave) per 100,000 population (1987 Office of Mental Health Update) and previously reported recidivism rates (comparing data of Weinstein, Dipasquale, Winsor 37 to Anthony, Buell, Sharratt, et al. ~ and Minkofl~~ New York does have a large proportion of the nation's psychiatric inpatient beds (15.1% in 1982) and state and county mental hospital beds (18.9% in 1982) (1987 OMH Update).

Study of Recidivism Rates

The following study was conducted to determine whether different psychiatric hospitals, with similar populations, catchment areas, fiscal restraints, census pressures, staff compositions, and overarching goals, have different recidivism rates for their discharged patients. In order to minimize between-hospital differences, six psychiatric facilities operated by the New York State Office of Mental Health were surveyed. To maximize their similarities, only facilities serving urban populations were chosen. Five of the six were tightly clustered together geographically within New York City. The sixth served a similar urban patient catchment area, though separated considerably from the other five geographically. In addition, to provide insight into temporal trends and within hospital changes, these recidivism rates were calculated across a four-year period. Finally, large samples of discharged patients from throughout the targeted hospitals wereused in order to minimize individual patient differences, the impact of particularly good or particularly bad treatment programs associated with specific wards within the individual hospitals, and postdischarge treatment differences.

It was hoped that this study would indicate whether the particular hospital from which a patient is discharged affects the successful reintegration of the patient into the community. As discussed above, these hospitals were chosen to maximize their similarities, therefore the ability to generalize from their respective recidivism rates to other facilities in New York State or beyond may be limited. However, the intent of the study was to gain insight into techniques for assessing recidivism, and, in cases where significant differences between hospitals were found, to gain greater understanding


of system and facility factors that influence recidivism rates. These facility and systemwide factors may have broader generalizability. In case between-hospital differences were found, it was hoped insight might be gained into facility and systemwide factors that might affect recidivism. There are many other aspects of recidivism, and associated facility and systemwide variables, that might have been included in this study. Because this study proposes a prototype for examining such issues, they are not included here, but they should be the subject of future research.

Method

The data represent all discharges in a three-month period, January 1 through March 31, for each of four years (1983, 1984, 1985, and 1986) at each of six state psychiatric facilities. Apatient could appear more than oncein a given year and may appear in more than one year. Although this introduces a small amount of dependence within samples, this is necessary to fully describe the psychiatric facility. These data were sampled from a patient specific movement file, the New York State Department of Mental Hygiene Information Services, which records all patient transitions, including admissions and discharges. Basic demographic information was collected on each patient discharged, along with the length of time to the next readmission to any New York State psychiatric hospital, if one occurred, by January 1, 1987, the termination point of the study.

Two different techniques were used to assess whether the recidivism curves between hospitals were statistically different. The first used only the proportion of the discharged patients still in the community as of January 1, 1987. The test statistic was a chi-square test, with various decomposi- tions as described in Cochran and Cox a9 used to investigate specific statistical hypotheses. The second technique analyzed all the data that make up the recidivism curve by a Kruskal-WaUis test. 4~ A modified Bonferroni procedure was used for instances in which multiple comparisons were performed. 4~ The Kruskal-WaUis test (and the chi-square test) is traditionally performed on independent samples. When comparing a facility across years, a given individual may appear in multiple years. This would violate the assumption of independence traditionally associated with the Kruskal- Wallis test; however, if anything, this reduces the likelihood of finding significant differences and thus only reinforces the strength of any significant findings that are reported. Statistical significance was declared at the two-talledp < .05 level.

R e s u l t s

A total of 7,407 discharges met the entry criteria for the current study. Basic demographic data are presented on each facility for the four study years in Table 1. Significant differences among the hospitals are present on all four variables reported (p < .0001 in all instances), reflecting in part differences in each hospital's catchment area, though other factors may contribute to these differences. The within-hospital demographic data are remarkably stable across time, with a few notable exceptions. As shown in Table 1, Facility 3 demonstrated a significant increase in the ages of discharged patients (p < .05), whereas Facilities 2 and 4 showed significant changes in the ethnic composition of their discharges (p < . 01 in both instances). Also, the percentage of the patients unable to work due to their illness differed considerably at each facility, with only Facility 3 showing no change. In general, this variable increased over time at all facilities. No other variables examined showed consistent changes across time.

The time to rehospitalization (recidivism) graphs are shown in Figures 1A-D for the years 1983 to 1986, respectively. Significant differences exist each year among the facilities (p < .0001), and the differences among facilities remain significant even if the one non-New York City facility (Facility 6) is removed. The variation in recidivism curves may be partially attributable to the differences in demographic characteristics of the discharge cohorts and potentially to characteristics

194 The Journal of Mental HealthAdministration 23:2 Spring 1996

Table 1

Demographic Distributions Across Facilities

Year

1983 1984 1985 1986

Facility 1 N 242 221 236 200 Median age 32 34 33 33 Male (%) 67 67 72 70 White (%) 21 27 21 17 Unable to work due to illness (%) 30 82 81 68






associated with the facility. However, the relative position of the recidivism curves for each facility changes dramatically from one year to the next, even though the demographics of the discharge cohort within the facilities remain rather constant across the years. Figures 1A and 1B also indicate that recidivism tended to level off approximately 11/2 years after discharge, with between 40 and 60% of the discharged cohort not having been rehospitalized by that time.


Figure 1 Comparisons Among Six State Hospitals, 1983 to 1986

Y e a r - 1 9 8 3 F i g u r e 1 A

Percent in Communi ty* lOe

90:

80

70:

60

50

40

30 0

-*- Facility 1 Jr Facility 2 -~ Facil i ty 3

, -~ Facility 4 -)(- Facil i ty 5 ~- Facil i ty 6

2 0 0 4 0 0 6 0 0 8 0 0 1 0 0 0 1 2 0 0 1 4 0 0

Days Following Discharge

Y e a r - 1 9 8 5

Percent in Community* 100}

F i g u r e 1 C

: ~Fac i l i t y 1 -I-Facil i ty 2 -)lSFacility 3

9 0

8 0

7 0

6 0

5 0

4 0 -

3 0 i , I i i i O 1 0 0 2 0 0 3 0 0 4 0 0 5 0 0 6 0 0


700

* Community indicates no return to a state hospital.

For all years, the significant difference among the facilities was present as early as 30 days postdischarge. Additionally, the curves are very concordant, suggesting that the facilities order themselves early and do not change their relative ranking. The Kendall coefficients of concordance


Figure 1 Continued

Y e a r - 1 9 8 4 F igure 1B

Percent in Community* 10(3

90

80

70

60

50

40

30 0

"0" Facility 1 -4- Facility 2 ~ Facility 3

I I I I

200 400 600 80O 100O


Y e a r - 1 9 8 6 F i g u r e l D

Percent in Communi ty*

lUL '~^- - - - 'Fac i l i t y 1 -I-Facil i ty 2 -)EFacility 3 n e ~ 90 ~ = ) ~ Facility 5 "~" Facility 6

80

70

60

50

L,,, I I I I I 0 50 100 150 200 250 300


* Community indicates no return to a state hospital.

for each year are 1983 = .69, 1984 = .87, 1985 = .86, and 1987 = .91. All coefficients are significant at t h e p < .0001 level. A coefficient of 1 would be a perfect ordering, with no change in ranks in

time. These results indicate that the length of time recidivism rates need be followed to detect


Table 2 Comparisons Between Facilities by Year of Discharge

1983

1985

1984 2 2 3 3 1 4 * *

4 1 * * 5 * * * * * 5 * *

6 2 3 1 4 6 2

1986 2 2 3 5 * 1 * * 3 * *

5 * * 4 * * 4 * * 1 * *

6 2 3 1 5 6 2

3 4 1

5 3 4

Note: Asterisks indicate significance at the .05 level, where p values have been corrected using a modified Bonfe~roni procedure. 41 For each year, facilities are listed vertically top to bottom and horizontally left to right from lowest overall recidivism rate to highest recidivism rate.

between-facility differences is relatively brief (e.g., 30 clays) when relatively large discharged patient sample sizes of this sort are used. However, because some variation remains, definitive statements regarding a particular treatment program at such an early point need to be made with caution.

Using a more extended time period encompassing all the data, facility differences can be tested more fully. All pairwise tests were performed between individual facilities using the entire time span for which data were available; p values were corrected to take into account the possibility that with a large number of comparisons, a certain number would be found to be significantly different by chance (see Methods). The results are shown in Table 2. Across all four years, Facility 6 had the lowest recidivism rate for its discharges, although the number and identity of facilities from which it significantly differed varied from year to year. At the other end of the spectrum, as shown in Table 2, for two (1983 and 1984) of the four years, Facility 5 had the highest recidivism rates, and for 1983 it was significantly different from all five other facilities, and in 1984 from four of the five.

However, the most important aspect of Table 2 is the systemic picture it provides of the differences between facilities and the considerable variation it shows in the pattern of differences from year to year. In 1983, one facility (Facility 5) clearly stands out from all the others, whereas the other five facilities are sufficiently similar in their recidivism rates to show no significant difference between their recidivism curves for that year. This pattern changes considerably in 1984. Although one facility (Facility 5) still distinguishes itself from most (4 of 5) of the other facilities, thereis now considerable interfacility variation, with every facility now statistically distinguishable from at least one other facility (with 8 of the 15 possible comparisons showing a statistically significant difference). In particular, Facilities 6 and 2 are now statistically significantly different from a number (three) of the other facilities, a pattern that will remain for the 1985 and 1986 discharge cohorts. The systemwide variation continues in 1985 and 1986, with five of the six in 1985 and all six in 1986 showing statistically significant differences in their recidivism rates from at least one other facility. Furthermore, as reflected in the relatively high Kendall coefficient of concordance for the three years


of 1984, 1985, and 1986, these systemwide differences show up relatively early and remain very consistent no matter what point following discharge is examined. Inasmuch as these facilities were selected for study because of their similarities, these results demonstrate that recidivism rates are a relatively fluid phenomenon, with differences between facilities showing up relatively soon after the discharge of the patient cohorts, showing some variability depending on what period of time after discharge is examined, but showing pronounced differences from year to year.

In order to understand within-hospital differences, the results in Figures 1A-D were replotted to show the progression of the recidivism curves across the four study years for each facility. Figures 2A-F plot the recidivism curves for 270 days after discharge for Facilities 1 through 6, respectively, Significant year-to-year variability is observed in Facilities 1 (p = .02), 2 (p = .01), an d 5 (p < .0001 ). The dramatic and abrupt changein recidivism curves between 1984 and 1985 in Facility 5 is obvious. It should be noted that a maj or change in hospital administration and staffing tookplace from March through July of 1984 following widely reported allegations of patient abuse and fiscal irregularities, 42a7 and the shift in recidivism rates may be attributable to these changes. However, other causes cannot be conclusively ruled out. In Facilities 1 and 2, a precipitous fallin recidivism curves occurred between 1985 and 1986. Similar, though not statistically significant, falls are seen in Facilities 3 and 4. A widely reported incident 4856 in the summer of 1986 that took place in the geographic locale of Facilities 1-5 was frequently cited as dramatically effecting admission and discharge policies in the area in the following months,5761 and this event may be reflected in the changes in recidivism curves. Finally, for Facility 1 there was a significant decrease in recidivism rates for the prior years (1983- 1985), a period of time when the facility regained JCAHO accreditation and was under constant scrutiny from an ongoing class action lawsuit challenging care and conditions at the facility.

D i s c u s s i o n

Using Recidivism Rates to Assess Effectiveness

The use of recidivism rates to assess the effectiveness of a mental health treatment program is a complex and potentially easy-to-misapply methodology. This study emphasizes the danger of relying on a reported recidivism rate to reach conclusions about a program based on a single point in time (e.g., six months after discharge), a single recidivism curve (e.g., a series of recidivism rates for a single cohort from a single facility in a single year), or a series of recidivism curves for a treatment program without comparing it to similarly situated programs. More succinctly, a re- searcher should be very cautious in attempting to conclude anything where a series of recidivism curves taken over a period of time and placed in comparison to those of similar treatment programs have not been provided. Prior studies suggest that many factors influence recidivism in addition to the nature and quality of the treatment program and that before concluding anything about the treatment program, these other factors need to be controlled for.

However, by examining recidivism curves across time, and by providing a similar series of recidivism curves for similarly situated treatment programs, an accurate picture of the performance of a mental health treatment program in particular, and a mental health system in general, can be obtained. Such an approach indicates how a program is performing over time and in comparison to similar programs.

For example, an examination of data from several points of time after discharge reveals a relatively wide range of recidivism rates. Collapsing data from all six hospitals and all four years, the range for three months after discharge is 14-43% (M = 30.1, SD = 8.3), for six months it is 22-50% (M = 38.0, SD = 7.9), for one year it is 32-57% (M = 46.2, SD = 7.6), and for two years it is 43-64% (M = 54.4, SD = 7.3). Relying on only one of these data points in assessing the effectiveness of a given hospital can lead to mistaken conclusions. Thus, whereas Facility 6 in 1985


E

r ~

o

t ~

o r~

0 L)

E

u.

, I/

r ~ ~ ~

E

g,

o o

o ~ ~ ~ ~' ~, ~o

m --

=

A

o

D

o

O

, f / ; H

i I -,

o

o~ n, i

ol

o

o

o

,i j/ ~ g, N

o ~ o o o o o ~~

appears to have a particularly low recidivism rate (18%) at three months after discharge, this same group of cohorts at the 18 months after discharge period show this facility to have a recidivism rate (47%) in the midrange for all hospitals across all years, and particularly for this given year. This suggests the importance of examining recidivism rates at more than a single point in time following discharge by constructing a recidivism curve for the program over an extended period of time. This will show whether a program, although it appears to be immediately successful in diminishing the likelihood that a discharged client will be rehospitalized, over the long term is not showing any pronounced success. Alternatively, another program, having adopted a philosophy that it is best to return the client to the community as quickly as possible with a minimum of disruption to that client's life, may consider a high number of immediate readmissions acceptable, provided a large number of clients are successfully reintegrated into the community. This program would show a relatively high recidivism rate in the immediate period following discharge, but with a recidivism curve that quickly flattens out after this initial period and remains relatively stable across time. If only rehospitalizations occurring shortly after discharge are examined, this program might be labeled a failure. However, by looking at the extended performance of this program, a very different conclusion might be reached. As an example of this, as shown in Fig. l-A, Facility 1 in 1983 had the second highest recidivism rate three months after discharge but at approximately four years after discharge was virtually tied for the second lowest recidivism rate and over this entire period of time was significantly different only from Facility 5.

Similarly, examining only a single recidivism curve for a facility can lead to mistaken conclusions. For example, Facility 5 showed almost the exact same recidivism curves for 1983 and 1984, but the curves for 1985 and 1986, although showing the same shape, generally reflected a recidivism rate 10% lower no matter what time after discharge was examined. Basing an assessment of the hospital on either the 1983 or 1984 curve alone might lead to a conclusion that the programming at this facility was relatively ineffective. In contrast, the recidivism curves for Facility 5 in 1985 and 1986 more closely approximate the midrange for all six hospitals. By studying all four curves in Figure 2-E, it becomes apparent that something occurred between 1984 and 1985 with a pronounced impact on the hospital. As noted earlier, this abrupt shift can probably be attributed to the major upheaval and relocation in the hospital administration and staff that took place from March through July of 1984 following widely reported allegations of patient abuse and fiscal irregularities. Examining only a single recidivism curve would have indicated none of these changes over time, or suggested that events internal to a hospital, but ostensibly unrelated to the programs themselves, would have such an impact on recidivism rates.

Finally, examining the recidivism rates and/or recidivism curves of a single treatment program, without comparing them to similarly situated programs, also leads to mistaken conclusions. An apparent slippage may be the result of factors external to the hospital over which the hospital can exert no or little control. For example, the New York City facilities (1-5) appeared to be showing gradual reductions in their recidivism rates over time from 1983-85, or were at least maintaining their prior recidivism rates. However, three (Facilities 1-3) of the five showed precipitous drops in their recidivism curves in 1986 (see Figures 2-A, 2-B, 2-C), with one of them (Facility 1) having the lowest overall recidivism rate for this year (see Figure l-D) and being significantly lower than three other facilities (see Table 2), after showing an overall recidivism rate in 1985 that placed it in the midrange of all facilities for 1985 (see Figure 1-C and Table 2). Examining the recidivism curves for any of those three facilities in isolation in 1986 would suggest that their programs had taken a sudden turn for the worse. However, as noted earlier, it is likely that the widely reported incident occurring in the summer of 1986 that tookplace in the geographic locale of New York City facilities may have been primarily responsible for these shifts. This explanation is further reinforced by the fact that the non-New York City facility continues to show an improvement in its recidivism curve in 1986 (see Figure 2-F).


Baseline Rates In general, the range of recidivismrates for these six facilities are comparable to therates reported

in other studies. This is somewhat surprising for two reasons. First, most previously reported recidivism rates for similar mental health programs considerably predated the time period examined in this study. Current research suggests that patients being admitted to state psychiatric centers today tend to be sicker, more hostile, and more difficult to treat than their predecessors. 3 This creates an expectation that today's facilities would experience a higher recidivism rate. However, this influence may be counterbalanced by the greater chronicity earlier patients developed as a result of long-term institutional treatment. Second, because the level of resources available for community care and follow-up in New York City are generally considered limited, it might be expected that NYC facilities would have considerably higher recidivism rates than other programs treating individuals with mental illness. Both points are underscored by the finding that the percentage of discharged patients described as unable to work increased in all six facilities across all years. The resemblance to other reported recidivism rates may suggest that recidivism rates are driven by a multitude of factors beyond simply the care provided within a particular treatment program and community and the nature of the patient population (at least as found within state psychiatric centers). The relative stability of recidivism rates of this study is shown by the similarities at the three-month and the six-month intervals when the six facilities are combined across the four years of this study (F = .48, p = .697).

Between-Hospital Differences In addition to these systematic patterns, there were significant differences between hospitals. For

example, in general the recidivism rates for the five New York City psychiatric hospitals were higher than for the one, non-New York City psychiatric hospital, even though it was also located in an urban area. Furthermore, as discussed above, this non-New York City facility showed a distinct reduction of its recidivism rates from 1985 to 1986, in contrast to the other five facilities, emphasizing how a localized event or series of events will not necessarily reverberate equally throughout a given system.

Facility 2 also had a relatively low recidivism rate. At the other end of the spectrum, Facility 5 had the highest recidivism rates for 1983 and 1984. However, although in 1985 Facility 5 had the second highest recidivism rate, by 1986 Facility 5 had the third lowest recidivism rate. Thus, although there is some fluctuation in the ranldngs of the facilities' recidivism rates at various points in time and across years, certain facilities do tend to have relatively distinctive recidivism rates.

In addition, the differences in the shapes of the recidivism curves for the various hospitals should be noted. For example, Facilities 2 and 6 showed a relatively linear progression in their recidivism curves over time, while the other four facilities all showed a much more curvilinear progression. Interestingly, these two facilities consistently showed the lowest recidivism rates, regardless of the length of time since discharge or the study year. Comparing the six graphs in Figure 2, it appears that the slopes of the recidivism curves for all years and all facilities are relatively similar at a point following 30-40 days after discharge. One potential explanation for the shape of these curves is that the initial 30-40 days are the most crucial in determining the likelihood of a discharged psychiatric patient continuing to reside in the community. However, caution must be exercised when deriving conclusions about individuals from group data--in other words, one must avoid the danger of the so-called ecological fallacy. 62

Implications for Mental Health Services Delivery The approach described in this study has at least three major uses for a policy maker. First, as

discussed, at the macrolevel it can be used to monitor and compare facilities within a system. Second, at the microlevel it can be helpful in deciding where to send an individual needing inpatient care.


For example, if the individual has strong community ties, it may be preferable to select a facility that appears to use an early discharge of patients in order to quickly reconnect the patient with his or her community resources. Third, where a programmatic change has been instituted, this approach suggests how much time is needed and what kind of factors need to be taken into account before an evaluation, relying on recidivism rates, of that programmatic change can be made with confidence.

Furthermore, enhancements to the approach described here might increase the ability to detect and understand differences in recidivism rates. In particular, identifying subpopulations within individual facilities and determining to whose care subgroups of the patient population were discharged (e.g., placed with family, a community residence, a group home, a shelter for the homeless) could further sharpen this analysis. Similarly, controlling for the length of time the patient was in the hospital prior to discharge or the number of prior admissions the individual had to a psychiatric facility, the two classic predictors of recidivism rate, might also increase the ability to identify and monitor facility differences in recidivism rates. However, it should be kept in mind that the pur- pose of this study was not to determine whether Facility A was better than Facility B, but rather to determine whether recidivism rates can be used as a general analytic technique. Adding the enhancements suggested above would help address questions regarding the quality of care within an individual facility.

A potential limitation of this study is that it focused exclusively on the recidivism rates for adult patients within state psychiatric facilities located in urban areas. It is possible that different patient populations (e.g., juveniles), different psychiatric systems (e.g., private facilities, wards associated with general hospitals, city-operated facilities), or different locales (e.g., rural, suburban) might show different recidivism patterns. However, it should be noted that the focus of this study has tended to be the target of the greatest concern when the recidivism of the mentally ill is discussed.

A second possible limitation of this study is associated with the definition of recidivism as a return to a state psychiatric facility. Clearly there are numerous other institutions that can become involved when a discharged psychiatric patient suffers a relapse (e.g., jails, private psychiatric facilities, shelters for the homeless). To the extent that there is a considerable variation in where discharged patients are reinstitutionalized within the general catchment area of the various facilities studied here, the results of this study may be skewed. Again, however, it was believed that the similarities in the facilities selected minimized any between-hospital differences.

Finally, recidivism rates tend to focus only on a general lack of quality within a facility while providing little information about what constitutes quality care. In other words, this study examines the failures and only indirectly the successes. It is also necessary and appropriate to evaluate the degree of adjustment of the discharged patient in the community and to identify factors that contribute to that adjustment. However, such a complex study is beyond the scope of this project. It is believed th at the approach described in this study, by providing insight into what hinders success, allows the policy maker to make reasonable estimates and draw appropriate conclusions about what might contribute to successful (i.e., nonrecidivist) discharges into the community.

This study suggests that perhaps the most appropriate use of recidivism rates to assess programmatic effectiveness is at the systemic level where a number of nonprogrammatic factors can be taken into account. However, even at this level the nature of the analysis will differ with the question asked. For example, if interest is focused on whether the recidivism rate for any facility stands out from the system as a whole, this can be demonstrated relatively quickly (e.g., 30-60 days postdischarge). However, if the intent is to compare a single specific facility to a number of other facilities individually, then a much longer period of time is needed to detect statistically significant differences. Furthermore, numerous nonprogrammatic factors that can influence recidivism rates can be detected only over a relatively long time and across a number of comparisons.

Thus analysts of a statepsychiatric hospital system, for example, can run a short-term recidivism study to generate a statewide perspective. However, when analyzing a single hospital, considerably more caution should be exercised in making determinations based on recidivism rates for short


periods of time after discharge. In other words, where recidivism rates are being used to gauge the quality of care provided within a facility (e.g., in determining whether the facility should qualify for fiscal reimbursement), using a relatively short time span to gauge recidivism in a large number of facilities, while indicating heterogeneity versus homogeneity in the system, will not suffice to determine the effectiveness of that program with a small probability of error.

Although this study did not focus on absolute levels of recidivism rates, the rates reported here may be used as a baseline for comparisons with other psychiatric facilities. Considering the well- documented nature of the problems faced by state psychiatric facilities located in the New York City region, other facility directors should probably be concerned if their recidivism rates are considerably higher than that of these facilities. However, such directors should also recognize theimportance of using their own facility's performance over time as a control and comparing that performance with similarly situated facilities.

In summary, these findings suggest that recidivismrates, or more appropriately recidivism curves, can be used to conduct comparisons between treatment programs, but only with great caution and by incorporating more data than most researchers have used in the past. Of particular concern is where a single recidivism rate for a single program is used to generate far-reaching conclusions about the effectiveness of that program. Any interpretation of the recidivism rate of the facility will require an assessment of the discharge philosophy of the facility, the length of time from discharge being examined, the setting in which the facility is operating, the direction the recidivism curve is moving over time, events occurring in the surrounding community that may affect this process, or events occurring within the program that are unrelated to the treatment itself.

It is apparent that recidivism rates can provide important information for individuals managing mental health programs and the mental health system. Such data suggest the diversity in a system while helping to identify the impact of significant events. The data reported here also suggest that while diverse factors operate on the mental health system, the administration of a particular facility does appear to have an impact on recidivism rates even within a relatively monolithic state system, although perhaps not to the extent commonly believed.

References 1. gobitscher, J: The Powers of Psychiatry. New York: Houghton Mifflin, 1980. 2. Committee on Government Operations: From Back Wards to Backstreets: The Failure of the Federal Government in

Providing Services for the Mentally Ill. House Report 100-541. Washington, DC: U.S. Government Printing Office, 1988.

3. Kiesler CA, Sibulkin AE: Mental Hospitalization: Myths and Facts About a National Crisis. Newbury Park CA: Sage, 1987.

4. Fisher WH, Pierce GL, Appelbaum PS: How flexible are our civil commitment statutes? Hospital and Community Psychiatry 1988; 39(7):711-712.

5. Seling MJ, Johnson GW: A bridge to the community for extended-care state hospital patients. Hospital and Community Psychiatry 1990; 41(2):180-183.

6. Waiters J: Making it easier to commit the mentally ill. Governing 1990; 3(6):60-61. 7. Lipton FR, Nutt S, Sabatini A: Housing the homeless mentally ill: Alongitudinal study of a treatment approach. Hospital

and Community Psychiatry 1988; 39:40-45. 8. Essock SM, Goldman HH: Prospective payment in retrospect American Psychologist 1989; 44(9):1237-1239. 9. Kiesler CA, Morton TL: Keeping an open mind: Arejoinder. American Psychologist 1989; 44(9):1239-1241.

10. Kiesler CA, Simpkins C, Morton T: Predicting length of hospital stay for psychiatric patients. Hospital and Community Psychiatry 1990; 41 (2):149-154.

11. Schiunar AP, Rothbard AB: Evaluation questions for Philadelphia's capitation plan for mental health services. Hospital and Community Psychiatry 1989; 40(7):681-683.

12. Baehrach LL: Planning rnentalhealth services for ehronicpatients. Hospital and Community Psychiatry 1979; 30:387- 393. 13. Fernandez GA, Nygard S: Impact of involuntary outpatient commitment on the revolving-door syndrome in Norlla

Carolina. Hospital and Community Psychiatry 1990; 41 (9):1001-1004. 14. General Accounting Office: Returning the Mentally Disabled to the Community: Government Needs to Do More.

Washington, DC: U.S. Government Printing Office, 1977.


15. Morrissey JP, Goldman HH: Cycles of reform in the care of the daronically mentally ill. Hospital and Community Psychiatry 1984; 35(8):785-793.

16. Weinstein AS, Cohen M: Young chronic patients and changes in the state hospital population. Hospital and Community Psychiatry 1984; 35(6):595-600.

17. Bond GR, Miller LD, Krurnwied RD, et al.: Assertive case management in three CMHCs: A controlled study. Hospital and Community Psychiatry 1988; 39(4):411-418.

18. Cutler DL: Clinical care update: The chronically mentally ill. Community Mental Health Journal 1985; 21:3-13. 19. Marcos LR: Who profits from deinstitutionalization? Hospital and Community Psychiatry 1989; 40(12):1221. 20. McRae J, Higgins M, Lyean C, et al.: What happens to patients after five years of intensive ease management stops?

Hospital and Community Psychiatry 1990; 41 (2):175-179. 21. Surles RC, McGurrin MC: Incxeased use of psychiatric emergency services by young chronic mentally ill patients.

Hospital and Community Psychiatry 1987; 38(4):401-405. 22. B either JR: Are jails replacing the mental health system for the homeless mentally ill? Community Mental Health Journal

1988; 24(3):185-195. 23. Caton CLM: The new chronic patient and the system of community care. Hospital and Community Psychiatry 1981;

32(7):475-478. 24. Harris M, Bachrach LL: Perspectives on homeless mentally ill women. Hospital and Community Psychiatry 1990;

41(23):253-254. 25. Okie S: Hospitals to be required to provide care statistics. Washington Post 1989; (October 1), p. A3. 26. Barter JT: Accreditation surveys: Nit-picking or quality seeking? Hospital and Community Psychiatry 1988; 39(7):707. 27. Hadley TR, MeGurrin MC: Accxeditafion, certification, and the quality of care in state hospitals. Hospital and Community

Psychiatry 1988; 39(7):739-742. 28. Anthony WA, Buell G J, Sharratt S, et al.: Efficacy of psychiatric rehabilitation. Psychological Bulletin 1972; 78:447-456. 29. Craig TJ, Kline NS: Factors associated with recidivism: Implications for a community support system. Community Support

Service Journal 1978; 2(2):1-4. 30. Minkoff K: Amap of chronic mental patients. In: Talbott J. (Ed.): The Chronic Mental Patient. Washington, DC: American

Psychiatric Association, 1978. 31. Erickson RC, Paige AB: Fallacies in using length-of-stay and returnrates as measures of success. Hospitaland Community

Psychiatry 1973; 24:559-561. 32. Winston A, Pardes H, Papernik DS, et al.: Aftercare of psychiatric patients and its relation to rehespitalization. Hospital

and Community Psychiatry 1977; 28(2):118-121. 33. Byers ES, Cohen S, Harshbarger DD: Impact of aftercare services on recidivism of mental hospital patients. Community

Mental Health Journal 1978; 14:26-34. 34. Hargreaves WA, Glick ID, Drues J, et al.: Short vs long hospitalization: A prospective controlled study. Archives of

General Psychiatry 1977; 34:305-311. 35. Leaf P: Patients released after Wyatt: Where did they go? Hospital and Community Psychiatry 1977; 28:366-369. 36. Strauss JS, Carpenter WT: Prediction of outcome in schizophrenia: Five-year outcome and its predictors. Archives of

GeneralPsychiatry 1977; 34:159-163. 37. Weinstein AS, Dipasquale D, Winsor F: Relationships between length of stay in and out of the New York State mental

hospitals. American Journal of Psychiatry 1973; 130(8):904-909. 38. Surles RC: Data from presentation to New York State Mental Health Services Council, Albany, 1988. 39. Cochran WG, Cox GM: Experimental Designs. New York: Wiley, 1957. 40. Siegel S, Castellan NJ: Nonparametric Statistics for the Behavioral Sciences. New York: McGraw-Hill, 1988. 41. Simes ILl: An improved Bonferroni procedure for multiple test of significance. Biometrika 1986; 73:751-754. 42. Finder A, Roberts K: Creedmoor's violent ways. The New York 1lines 1984; (May 20), p. 6. 43. Shenou P: Mental patients reported abused at Queens center. The New York Times 1984; (May 14), p. A1. 44. Shenon P: Creedmoor aide indicted over death. The New York Times 1984; (May 31), p. B5. 45. Shenon P: Fear and brutality in a Creedmoor ward. The New York Times 1984; (June 18), p. A1. 46. Shenou P: After inquiry at Creedmoor, 10 resign or face ddsmissal. The New York Times 1984; (June 19), p. B3. 47. Sullivan R: Abuse cited in death of Creedmoor patient. The New York Times 1984; (April 25), p. B24. 48. Douglas CC, Counelly M: City aides assail release of patient in ferry attack. The New York Times 1986; (July 13), p. 6. 49. Freedman SG: Neighbors noted traits of violence in suspect. The New York Times 1986; (July 9), p. B2. 50. McFadden RD: Man with sword kills 2 and wounds 9 on S.I. ferry. The New York I~mes 1986; (July 8), p. A1. 51. Purdum TS: Police assign teams to all ferries following sword slaying of two. The New York Times 1986; (July 9), p. A1. 52. Sullivan R: Hospital outlines care of slashing suspect. The New York Times 1986; (July 10), p. B1. 53. Sullivan R: Doctors had tried to hospitalize suspect in slashing on ferryboat. The New York Times 1986; (July 11 ), p. A1. 54. Sullivan R: Ferry slashing report assails hospital for freeing suspect. The New York Times 1986; (July 12), p. 1. 55. Sullivan R: State faults ferry suspect's release. The New York Times 1986; (July 19), p. 31. 56. Sullivan R: State panel criticizes emergency psychiatric care. The New York Times 1986; (August 3), p. 1. 57. Delusions about mental health. The New York Times 1986; (August 1), p. A26.


58. Sullivan R: Psychiatric treatment staging since ferry killings. The New York Times 1986; (July 25), p. A1. 59. Sullivan R: City acts on crowding in its psyehialric wards. The New York Times 1986; (August 14), p. B3. 60. Sullivan R: Hospitals agree to help city treat disturbed. The New York Times 1986; (August 21), p. B1. 61. Sullivan R: State asking hospitals to add psychiatric beds. The New York Times 1986; (November 14), p. B3. 62. Goodman LA: Some alternatives to ecological ecrrelafion. American Journal of Sociology 1959; 64:610-625.

206 The Journal o f Mental Health Administration 23:2 Spring 1996

Documents

Methodological advances in the use of recidivism rates to assess mental health treatment programs