On the measurement of technical efficiency in the public sector

~ : : . 7 I

i: (ll E L S E V I E R European Journal of Operational Research 90 (1996) 553-565

EUROPEAN JOURNAL

OF OPERATIONAL RESEARCH

Theory and Methodology

On the measurement of technical efficiency in the public sector

John Ruggiero Department of Economics and Finance, University of Dayton, Dayton, OH 45469-2240, USA

Received November 1993; revised September 1994

Abstract

Existing measures of technical inefficiency obtained through linear programming models in the public sector do not properly control for environmental variables that affect production. It will be shown that the consequences of not controlling for these fixed factors are biased estimates of technical efficiency. This paper extends the mathematical programming approach to frontier estimation known as Data Envelopment Analysis to allow for environmental variables. This modified model will be then contrasted with the existing model that purportedly controls for exogenous factors to measure public sector efficiency with simulated data. The results provide evidence that the existing Data Envelopment Analysis model will overestimate the level of technical inefficiency and that the modified model developed in this paper does a better job controlling for exogenous factors. The modified model is also applied to analyze the technical efficiency of school districts.

Keywords: Efficiency; Data Envelopment Analysis; Public sector production; Environmental variables

1. Introduction

Based on the work of Farrell [14], a body of l i terature has been developed to measure the economic performance of individual decision making units. In particular, the basis for analyzing performance is the measurement of technical efficiency. Following Koopmans [20], a feasible inpu t -ou tpu t vector is said to be technically efficient if it is technologically impossible to increase any output a n d / o r reduce any input without si- multaneously reducing another output a n d / o r increasing one other input. Debreu [11] and Farrell [14] provided an index of the degree of technical efficiency with Debreu ' s 'coefficient of resource utilization', computed as one minus maximum equiproport ional reduction in all inputs consistent with equivalent production of existing out-

puts. (Alternatively, output-oriented models provide efficiency indices based on output augmentation holding inputs constant. Also, for a discussion of the differences between the D e b r e u - F a r - rell measure and the Koopmans measure, see Lovell [21].) With this measure an index is cre- ated that allows comparison of producing units in terms of its overall technical efficiency; an index value of one implies technically efficient production while a value less than one implies deviation from the frontier.

The pioneering work on technical inefficiency of Debreu and FarreU has received renewed interest in the last decade with the development of Data Envelopment Analysis (DEA), a mathematical programming approach to frontier estimation pioneered in Charnes, Cooper and Rhodes [10] and extended in Banker, Charnes and Cooper [2].

0377-2217/96/$15.00 © 1996 Elsevier Science B.V. All rights reserved SSDI 0377-2217(94)00346-7

554 J. Ruggiero / European Journal of Operational Research 90 (1996) 553-565

DEA is a programming technique that extends the theoretical discussion of technical efficiency of Debreu, Farrell, and Koopmans into direct measurability. DEA is empirical in nature; it en- velops the observed data to determine a best- practice frontier. The DEA approach has become popular in evaluating technical efficiency of local governmental authorities in the public sector because it easily handles the multiple outputs char- acteristic of public sector production, is non-para- metric, and does not require input price data, which is often difficult to measure accurately in the public sector. (There are disadvantages, however. Since DEA is non-stochastic, the results are sensitive to measurement error and incorrect variable selection. For a discussion, see Seiford and Thrall [24] and Banker, Gadh and Gorr [3].) A popular application is the analysis of school districts; school district performance has been analyzed to determine the extent of relative technical inefficiency by Bessent and Bessent [7], Bessent, Bessent, Kennington and Reagan [8], Jessoon, Mayston and Smith [18], and Fare, Grosskopf and Weber [13].

The applicability of DEA, however, depends crucially on the underlying production process. It is essential that the model be consistent with known properties of the production process. Pro- duction in the public sector has been modelled by Bradford, Malt and Oates [9] as a two-stage process where the intermediate output is determined in the first stage by a production process involv- ing only discretionary (i.e., variable and control- lable by the decision making unit) inputs. In the second stage, final outcomes are determined by the level of the intermediate output and by environmental (i.e., fixed)variables. Empirical studies of public sector production support this theory; environmental variables have a substantial impact on the outcomes that are provided. A good exam- pie is the provision of the fire services (measured perhaps by the number of lives saved and/or the dollar amount of property damage prevented) by local communities. We would expect that a community with a higher proportion of brick houses would be able to provide a higher level of services than a community consisting primarily of old wood houses, assuming the same usage of inputs (fire

fighters, fire engines, etc.). In this case residential structure is exogenous to the decisions of the fire departments.

Another prominent example is the provision of educational services. School districts use labor (teachers) and capital (computers, facilities, etc.) to provide instructions in mathematics, reading and other content areas. In the Bradford, Malt and Oates framework, these actual outputs are not of primary interest to the community; rather, residents are concerned with final outcomes (for example, test scores, drop-out rates, etc.). There is strong evidence of the influence that environmental factors such as parental background and socio-economic variables have on the transforma- tion of school inputs into the services desired by voters (see Hanushek [16,17]). (As pointed out by an anonymous referee, environmental conditions could exist in a general theory of organizations. This suggests that the public sector DEA model developed in this paper could be extended to analyze other types of organizations).

Because environmental variables are exogenous to the production process it is necessary to modify the existing DEA model to properly control for these fixed factors. The early DEA studies analyzing school district performance have not. To solve this problem, Banker and Morey [4] introduced a DEA model that purportedly allows for exogenously fixed factors. As will be shown, however, this model may not be consistent with the underlying production process for the public sector. In this paper, I provide a description of public sector production technology and develop a modified DEA model that is consistent with this description. (An alternative approach com- bines DEA and tobit analyses to control for environmental effects. For an example, see McCarty and Yaisawarng [23]. This method requires a priori functional form specification of the relationship between environmental variables and the production frontier. The approach developed in this paper provides a flexible way to control for the environment in one stage.)

The basis for analyzing technical inefficiency is the production function developed in standard microeconomic theory. This production function for a private firm shows the maximum possible

J. Ruggiero / European Journal of Operational Research 90 (1996) 553-565 555

output that could be produced for given input levels. Production in the public sector, however, is affected by the presence of environmental factors. Accordingly, the production function in the public sector shows the maximum output possible for various levels of discretionary inputs, holding environmental variables constant (this parallels the distinction between the short-run in standard mi- croeconomics).

Banker and Morey [4] modified the measure of inefficiency obtained by removing the effect of fixed factors on the measured inefficiency level within the DEA model. Their model, however, may not restrict the reference set enough for the DMU under analysis to reflect the importance that environmental variables have on production. Consequently, an efficient DMU facing a harsh environment may be determined to be relatively inefficient because the reference set includes DMUs with more favorable environments, imply- ing that the Banker and Morey model may overestimate the level of technical inefficiency. This paper shows that this anomaly results from an inconsistency of the Banker and Morey (BM) model with public sector production theory. The main purpose of this paper will be to develope a modified DEA model that maintains consistency with known properties of public sector production. (Note: As pointed out by an anonymous referee, the model developed in this paper is an extension of the Banker and Morey [5] model that allows for categorical variables. See also Ka- makura [19]. The model developed here extends Banker and Morey [5] to allow continuous environmental variables. Because of this, the modified model is comparable to the Banker and Morey [4] model for exogenous inputs and not Banker and Morey [5]. For future references, the Banker and Morey model will refer to the model developed in Banker and Morey [4], i.e. the model that allows continuous exogenous variables.)

The rest of the paper is organized as follows. The next section presents a standard description of the technology facing the private firm when all inputs are variable. This description is then for- malized into the input-oriented DEA model developed by Banker, Charnes and Cooper [2]. The technology description is then extended to the

public sector and the Banker and Morey model is presented. It will be shown that this model may lead to biased estimates of inefficiency, capturing not only inefficiency but also the effect that the environment has on production. In Section 3, a modified DEA model is developed based on the public sector DEA postulates presented in Sec- tion 2. Section 4 uses simulated data to facilitate comparison between the Banker and Morey model and the newly developed modified model. The results support the hypothesis that the Banker and Morey model overestimates technical inefficiency. Further, the newly developed modified model appears to overcome identified weak- nesses of the BM model. Section 5 contains an empirical analysis of New York State school districts in school year 1990-91. In anticipation of the results, it is found that a significant amount of inefficiency exists in the provision of educational services. The last section concludes with directions for future research.

2. Description of the technology

2.1. Production without environmental inputs

Consider a production unit employing M discretionary inputs X - (x l , . . . , xM) ~ ~+M in the production of S outputs Y - (Yl,--., Ys) ~ Es. Then, given an implicit production function Y= f ( X ) showing the maximal amount of outputs that can be produced with a given set of inputs, the production possibility set can be defined as

T-- {(X, r ) : r < ~ f ( X ) } . (1)

An estimate of the frontier is provided based on the following DEA postulates (which are presented in Maindiratta [22]):

Postulate 1 (Convexity). If (X', Y') ~ T and (X", Y") ~ T, then for any scalar 0 ~ [0,1],

(OX' + ( 1 - O)X", OY' + (1 - O)Y") ~ T.

Postulate 2 (Monotonicity). (a) If (X, Y) ~ T and X'>~X, then (X' , Y ) ~ T.

(b) If (X, Y) ~ Tand Y' <~ Y, then (X, Y') ~ T.

556 ./. Ruggiero / European Journal of Operational Research 90 (1996) 553-565

Postulate 3 (Inclusion). The observed (Xj, Y) ~ T for all DMUs j = 1, . . . , n.

Postulate 4 (Minimum extrapolation). I f a production possibility se t T 1 satisfies Postulates 1, 2, and 3 above, then T 1 c_ T.

DEA estimates T among all production possibilities that satisfy convexity, monotonicity, and inclusion. This results in an estimate T 1 of the production possibility set T that is unique, closed, and of the form

T~ = {(X, Y) : X>~

E oixi , r <~ E oiYi, i i

where 0 i >1 0 are scalars and ~ 0 i "~ 11 . (2) i !

Min

s . t .

Based on T 1, Banker, Charnes and Cooper [2] have shown how the estimated frontier Tx can be used to determine the relative technical efficiency of producing units. Defining sff = (S~0 . . . . , S~0) and s o = (s~, . . . ,Ss0) as vectors of input and output slacks, respectively, that represent the ad- ditional amount of input reduction and output augmentation that is possible after all inputs have been proportionally reduced, the DEA program can be operationalized with the following linear programming model to evaluate the technical efficiency E0 of DMU0:

e0 - + E (3a) \ i = 1 r = l

F, - So = Yo, (3b) j = l

ojx + = e 0 x 0 , (3c1 j = l

0 r = 1, (3d) j = l

Oj>~O V j = l . . . . . n, (3e)

s o ,s~- >~ 0, (3f)

• > 0 is non-Archimedean. (3g)

Ignoring slacks, the above linear programming model determines the maximal proportional reduction in all inputs (constraint (3c)) possible holding output constant at the level of the DMU under analysis. As such, this model is considered to be input-oriented. Further, this model allows variable returns to scale via the convexity constraint (3d). Determination of the technical efficiency of all DMUs requires solving n linear programs (one for each DMU). The maximum possible value for E o is 1; this can be seen by noting that the DMU under analysis (DMU 0) is one of the n DMUs. This DEA model can be considered two-stage in the sense that the model first determines the maximum possible radial reduction (1 -E0* ) in all inputs from the observed level X o followed by any further input slack reductions and any output slack increments possible. (Therefore, this model is able to measure technical efficiency according to both the De- breu-Farrell definition and the Koopmans definition. It should be pointed out that the use of slacks is not universally accepted. For a discussion, see Lovell [21] and Ali and Seiford [1].) Reduction of the two-stages into a single model is achieved by the inclusion of the non-Archi- medean infinitesimal e that determines any existing slacks only after the Farrell-Debreu technical efficiency measure is calculated. (Any computa- tional problems that arise by using e can be solved by using a two-stage model; see Ali and Seiford [1].)

The solution of this model will contain information on the efficient production level for DMU 0. In particular, DMU 0 could achieve technical efficiency by reducing discretionary inputs to the level

xiEo =EoXio-S~o for all x i ~ X

and increasing all outputs to the level

Yreo = YrO + S~ for all Yr ~ Y"

For future purposes, we will denote the technically efficient level of production (or alternatively, the composite reference group) for DMU 0 as ( X ~ , Yo* ) where

x ; =- (X o . . . . . X ,o)

J. Ruggiero ~European Journal of Operational Research 90 (1996) 553-565 557

and

Yo*- (Y,~, .-- , Y~0)-

Because the outer envelope of the data is con- structed via a convexity constraint, this model allows variable returns to scale (VRS). The origi- nal model developed by Charnes, Cooper and Rhodes [10], following the work of Farrell, assumed constant returns to scale. The constant returns to scale model can be obtained from the above formulation by removing the convexity constraint (3d).

The measurement of technical efficiency is shown in Fig. 1. Referring to the diagram, assume that four decision making units (A, B, C, and G) employ one input X to produce one output Y. Based on the above postulates, the frontier is estimated as convex combinations of the observed production possibilities. This leads to the estimate of the frontier consisting of line segments AB and BC. In this case, inclusion of the convexity constraint (3d) leads to increasing returns to scale along AB and decreasing returns along BC. The VRS model presented above measures the technical efficiency of DMU G to be X I / X G, where the composite reference production possibility is labeled I. This measure can be considered to be a 'pure ' technical efficiency measure since it allows variable returns to scale. By assuming constant returns to scale in production, the technical efficiency of DMU G would be estimated to be X n / X G < X~/X G. It has been shown

Y !

C

I i /,'

y , H I/

by Banker, Charnes and Cooper [2], Fare, Grosskopf and Lovell [13] and Banker and Thrall [6] that the measure of inefficiency 1-E obtained from the solution to the constant returns to scale DEA model consists of not only technical but also scale inefficiency.

The DEA model presented above was developed to be consistent with the underlying production process for a firm that faces no environmental factors. However, since D EA has been primarily applied to the public sector, it is necessary to extend the above description to allow for exogenously fixed factors.

2.2. Production with environmental inputs

Maintaining the notation used above, suppose that each DMU uses discretionary inputs X = (x 1 . . . . . x M ) ~ R~ to produce outputs Y - (Yl . . . . . Ys) ~ ~s given exogenous inputs Z = (Zl . . . . . ZL) ~ RL+. TO operationalize D EA in this case the production possibility set is redefined for a given environment as

T(Z) - {(S, Y ) : Y ~ f ( S l Z ) } (4) and modify the DEA postulates:

P.1. (Convexity) VZ, if (X' , Y ' ) ~ T(Z) and (X", Y") ~ T(Z) , then for any scalar 0 ~ [0,1],

( o x ' + (1 - o ) x " , OY' + (1 - o)r" ) ~ T( Z) .

P.2. (Monotonicity) VZ, (a) I f (X, Y ) ~ T(Z) and X ' >~ X, then (X ' ,Y) T(Z). (b) I f (X, Y) ~ T(Z) and Y' <~ Y, then (X, Y') T(Z),

P.3. (Inclusion) VDMUs j = 1 . . . . . n, if DMUj faces Zj, then the observed ( X~, Y) ~ T( Z).

P.4. (Minimum extrapolation) I f a production possibility set TI(Z) satisfies postulates P.1-P.3 above, then TI(Z) _ T(Z),VZ.

Xlt X I XG X

Fig. 1. Measuremen t of technical efficiency. Assuming one input and one output , the D E A measure of efficiency is shown for D M U G.

To extend D EA to the public sector, it is necessary to posit the relationship between the environment and the production frontier. To do so the following postulate is introduced.

558 J. Ruggiero ~European Journal of Operational Research 90 (1996) 553-565

P.5. (Environmental effect) T(Z 1) __. T(Z 2) for all Z 1 -~< Z 2 .

Here, Z 1 ~< Z 2 implies that Z 1 is an environment no better than Z 2. In essence, this postulate allows the existence of nested production possibility sets and implies that a DMU with a more favorable environment should be able to produce at least as much output as any DMU with a less favorable environment (note, however, that this allows for equivalent production frontiers for different environments). This is consistent with the Bradford, Malt and Oates [9] two-stage public sector service provision model. (Actually, this generalizes their model, allowing environmental variables to enter the production process in the first-stage.) With these modifications it becomes clear that a given DMU should be compared only to DMUs with an environment that is at least as harsh as the one that it faces.

This implies that DEA should estimate T ( Z ) for all Z from all production possibilities that satisfy postulates P.1-P.5. This results in an estimate T2(Z),VZ, of the production possibility set T ( Z ) that is unique, closed, and of the form

= ((x , Y): Eoix,, EoiY , T2( Z) i i

where 0, >~ 0 are scalars,

#i = 1, and Z i < Z I . (5) E i 1

The effect of the environment on the production frontier is shown in Fig. 2. It is assumed that there are 7 DMUs producing one output Y with one input. In addition, it is assumed that DMUs A, B, and C face a relatively favorable environment compared to DMUs D, E, F, and G. As a result, DMUs A, B, and C are able to produce more output with a given set of inputs. Without controlling for the environment, the variable returns DEA model measures the level technical efficiency of DMU G as X H / X G. This is clearly inappropriate since the efficient projection for DMU G, point H, is not feasible given the environment that G faces. In addition, DMUs D, E, and F would be evaluated as technically inefficient, when in fact, they are operating on their

Y

c

z. " F

B r J / /

o

X X n X 1 XQ

Fig. 2. Technical efficiency in the public sector. DMU G, facing a similar learning environment as DMUs D, E, and F would be compared to point H in the variable returns model. However, this is not feasible.

Min

s.t.

associated frontier. To evaluate the efficiency of DMU G, it is necessary to compare G only to those DMUs with at least as harsh an environment as G faces. (As pointed out by an anonymous referee, an endogeneity problem may arise if the environmental variables are also causes of inefficiency. It will be assumed for purposes of this paper that this problem does not exist; it should, however, be the subject of further research).

Before developing a modified DEA model to control for exogenously fixed factors, the BM input-oriented model for multiple outputs will be presented. This input-oriented formulation to measure the technical efficiency of DMU o facing environmental factors Z 0 is given by the following linear programming problem:

P0 - e s/~ + s~0 (6a) t , i = l

0jYj - s o = Y0, (6b) j=l

n

E OjXi + s~ = poXo, (6c1 j=l

0jZ~ + s0 f = Z0, (6d) j=l

n

E Or = 1, (6e) j=l


0j>~0 Vj---1 . . . . , n , (6f)

s~-, s o , s f >t O, (6g)

> 0 is non-Archimedean, (6h)

where S0 f = (S~0 . . . . . s f0 ) represents slack on the fixed factors and is included to enrich the comparison set in the BM model.

The modification of the variable returns D E A model to obtain the BM model is the inclusion of a constraint (6d) on the level of the fixed factors. This modified D E A model requires only that the composite reference group (X0*, Y0*) does not have a more favorable environment then the D M U under analysis. However, this constraint may not be consistent with the technology of a public sector firm.

Proposition 1. The modified technical efficiency component Po for D M U o facing Z o obtained from the solution to BM will underestimate the level of technical efficiency if for the composite reference group determined from BM, ( X~ , Yo* ) q~ T( Zo).

This is proved by noting that restrictions are placed on the level of Z o for the composite reference group but not for the individual DMUs in the reference set. Thus, an optimal solution to the BM model for D M U 0 (X0*, Y0*) may include in the reference set DMUj where Zj > Z 0, which may violate postulate P.5. Consequently, without further restricting the reference set, a solution (X0*, Y0*) may not be feasible. This is best illus- t rated with an example.

Suppose three DMUs A,B, and C produce one output y using discretionary inputs x I and x 2 while facing environmental factor z. Further, suppose that all three DMUs are operating efficiently according to the production schedule shown in Table 1. Since D M U A faces the harsh- est environment (i.e., z is the lowest), it must use

Table 1 Hypothetical production data

DMU y x 1 x z z

A 10 10 10 20 B 10 8 8 30 C 10 1 1 40

more of both discretionary inputs to produce y = 10. As can be seen by the schedule, if the environment becomes more favorable then less of both inputs are required to produce the given level of output. By applying the BM model, we obtain an efficiency rating for D M U B of E B = 0.6875. This results because D M U c is allowed into the reference set. The solution for DMUB is

1 obtained when 0 A = 0 c = ~ resulting in production YB = 10 from inputs Xla =xzB = 5½ for the composite reference group. This is feasible under the BM assumptions since the composite reference group utilizes z * = 30 = z a . However, this contradicts the assumption that all DMUs are operating efficiently. At the same time, the efficiency ratings of 1 for A and C obtained are not biased since both (X:~, YA*) = ( X A, YA) and ( X ~ , Yc* ) = (Xc , Yc) are clearly feasible.

3. Developing a DEA model for the public sector

The hypothetical example presented above il- lustrates that the BM model may not be appropriate if the description of the technology for the public sector developed in this paper is accurate. Since the description is consistent with existing empirical findings on public sector production, it is important to develop a D E A model that will reflect the underlying production process. Insight into the development of the appropriate D E A model is provided from the description of the technology itself. We note that the efficient level (X0*, Y0* ) obtained from the solution of any D E A model must be feasible. Recognizing this, we can now state the modified input-oriented D E A model as:

Min

s.t.

' ;o) k i = l r = l

(7a)

o y , - So = go, (7b) j=a

OjXj + s~ = ~7oXo, (7c) j= l


L 0j = 1, (7d) )=1

0/>>.0 W ~ Z j < ~ Z o, (7e)

0 j = 0 W ~ Z j > Z o, (7 0

s~-,s o >/0, (7g)

e > 0 is non-Archimedean.

The restrictions necessary for an output-oriented formulation is similar. The appropriate modification for measuring technical efficiency should insure that the resultant composite reference group is feasible. The only way that this can be guaranteed is by constructing the reference set to include only those DMUs with at least as harsh an environment as the one that DMU 0 faces. This is achieved by constraining the multipliers to zero for DMUs with a more favorable environment (constraint (7f)). This leads to the following proposition:

Proposition 2. The technical efficiency rating ~1o determined from the modified DEA model is greater than or equal to the technical efficiency rating determined from the Banker-Morey model, Po.

This can easily be seen by considering that an optimal solution to the modified model is feasible in the BM model since the constraint on the fixed factors in the modified DEA model is binding in the BM model. Furthermore, the modified model is consistent with the underlying technology for the public sector that was developed in Section 2.2. The constraints of the modified model assure that the composite reference group's production is feasible.

4. A simulated analysis

To illustrate the potential biases that may result from using the BM model to measure technical efficiency, a simulation was performed assuming two discretionary factors of production, K and L, are used to produce one output Y. It was further assumed that production was influenced by one environmental variable Z. The production

function was specified as constant returns to scale with respect to the discretionary inputs K and L. In particular, it was assumed that

Y = K1/ZL1/2Z,

where Z acts as a Hicks neutral shift factor. This production function is consistent with the underlying technology developed in Section 2.2: DMUs with a more favorable environment are able to produce more output with the same level of inputs.

All inputs were randomly generated from a random uniform distribution for 250 observations according to the following intervals: K: (10, 30); L: (20, 40); Z: (1, 2). From these values, the efficient level of output was generated. The observed level of output was generated as ( 1 - 3') times the efficient level of output, i.e.,

yobs = (1 -- T ) K 1 / Z L 1 / z z .

Thus, the true level of technical inefficiency was known to be y, where y was generated from the following distribution:

y ~ IN(0,0.2036)I.

The value of y was further restricted by set- ting y equal to zero if the value of y generated was greater than 0.30. As a result, 37 out of the 250 DMUs were generated to be efficient; the average technical efficiency of all DMUs was generated to be 89%. To facilitate comparison between the two models, the constant returns to scale versions of both BM and the modified model were used. (Because the technology is character- ized by constant returns to scale (CRS), only the results from the CRS models are reported. An analysis of the variable VRS models was also performed; the results, which are similar to reported CRS results, are available upon request.)

In Table 2, a detailed analysis is presented for DMU100, which is known to be 16% inefficient. The BM CRS model estimated the technical efficiency to be 0.68, which underestimates the known technical efficiency of 0.84. Thus, the results suggest that DMU100 could proportionally reduce K and L by 32% to achieve technical efficiency.

Z Ruggiero/European Journal of Operational Research 90 (1996) 553-565

Table 2 Identification and description of the reference set for DMU10o ~ (Z = 1.6, known efficiency level = 0.84)

561

DEA model Estimated Referent 0j Level of the efficiency DMUs fixed factor

BM model 0.68 37 0.62 2.0 38 0.12 2.0

Modified model 0.84 161 0.91 1.6 198 0.01 1.6

a The results are obtained assuming CRS.

Since the t rue level of inefficiency is only 16%, however, this results in a project ion to a produc- t ion p lan that can not be feasible. The measure of efficiency ob ta ined via BM is con t amina t ed with the impact that the env i ronmen t has on produc- t ion. (Note that the BM VRS model overestima ted the level of inefficiency to be 0.22.) This is

Table 3 Evaluation of overall performance

not the case for the modif ied model: The efficiency index ob ta ined equa led the correct value.

Table 2 also identif ies the D M U s in the reference set and their associated weights 0 ob ta ined in the solut ion of each model . The BM model

allowed into the re ference set DMU37 ( Z = 2.0) and DMU38 ( Z = 2.0), bo th of which have a more

Criteria Known BM model Modified model

All districts: Number efficient 37 Number inefficient 213 Mean efficiency 0.89 (standard deviation) (0.09) Median efficiency 0.90 Correlation with true efficiency:

Pearson's coefficient 1.00 Spearman's coefficient 1.00

Inefficiency overestimated: Number of DMUs N.A. a I Average difference I N.A.

Inefficiency underestimated: Number of DMUs N.A. [ Average difference I N.A.

I Average difference I : BM estimates closer N.A.

(10 DMUs) Modified model estimates closer N.A.

(221 DMUs)

9 54 241 196

0.70 0.91 (0.15) (0.08) 0.69 0.92

0.44 0.96 0.41 0.96

227 0 0.21 N.A.

18 213 0.02 0.02

0.03 0.06

0.22 0.02

Inefficient districts: Mean efficiency 0.87 0.68 0.89 (standard deviation) (0.08) (0.15) (0.08) Mean efficiency 0.87 0,65 0.89 Correlation with true efficiency:

Pearson's coefficient 1.0 0,43 0.95 Spearman's coefficient 1.0 0.40 0.94

a N.A. = not applicable. Source: computations by author using the CRS model.

562 J. Ruggiero /European Journal of Operational Research 90 (1996) 553-565

favorable environment than the one DMU10 o faces (Z = 1.6). Notice that the level of the fixed factor for the composite reference group (Z * = 1.473) obtained from the BM model suggests that the composite reference group faces an environment as harsh as the one faced by DMU~00, even though both DMUs within the reference group face a better environment. (Note that this has implications for nonparametric estimation of returns to scale.) This reinforces Proposition 1; the restrictions placed on the reference set in the BM model may not be restrictive enough, leading to biased estimates of technical inefficiency. The modified CRS model (as well as the modified VRS model), on the other hand, correctly measured the efficiency of DMU~0 o to be 0.84. Inter- estingly, the reference sets from obtained from the BM model and the modified model are dis- joint. As will be shown, the results obtained for DMU100 are representative of the simulation results.

Table 3 reports the overall results obtained from the simulation. In general, the results suggest that the modified model outperforms the BM model. Confirming the results obtained for DMU10 o obtained above, a pattern of overestima- tion of inefficiency emerges. The BM model overestimated the level of inefficiency for 227 DMUs, with an average difference of 0.21. Unlike the BM model, the modified model did not overestimate inefficiency for any DMU. The reason this occurs, as discussed above, is the inappropriate treatment of fixed factors within the BM model. This result is highlighted by the identification of 28 districts that were known to be efficient as technically inefficient. The results of the analysis suggest that the modified model provides more conservative estimates of inefficiency, which was not surprising.

To evaluate the overall performance, correlation coefficients were calculated between the estimates and the known level of efficiency. Pearson's correlation coefficient suggests that the modified model's estimates are more highly correlated, achieving a correlation of 0.96 (compared to 0.44) for all DMUs used in the analysis. Spearman's rank correlation coefficient also suggests that the modified model does a better job; the rank corre-

lation (0.96) between the modified estimates of inefficiency and the known level are more than twice as high as the corresponding rank correlation for the BM model (0.41). (Note that a Wilcoxon signed-ranks test was also performed; the results, which are not reported, confirm the results reported in Table 3, that the modified model does a better job controlling for the environment. The BM model produced estimates closer to the true level in absolute terms for only 10 DMUs; on the other hand, the modified model was closer for 221 DMUs). The overall results are suggestive: without controlling for environmental factors that influence production technical efficiency results may be biased.

5. Empirical analysis

This section provides an empirical analysis of educational production for New York State school districts. Technical efficiency of 556 school districts in New York State for the school year 1990-91 is estimated using the variable returns to scale DEA model (7) presented above. (There were 691 major school districts in New York State. Due to missing observations, including New York City, the sample was limited to 556 observations for which all data were available.) This section describes measures, data sources, and results.

5.1. Variable definitions

A fundamental issue involved in any educational production analysis is the choice of outcome measures. The New York State Depart- ment of Education, as part of its 'Comprehensive Assessment Report ' (CAR), collects a variety of school 'outcome' measures each of which cap- tures a different dimension of student achieve- ment. Here, as in most educational production studies, outcomes are measured as the average score on standardized tests given to all students. In New York, the only comprehensive achieve- ment test required to be taken by all students is the Pupil Evaluation Program (PEP) tests given in third and sixth grades. Outcomes are measured


as the average test scores for a district in each of reading, mathematics, and social studies for sixth grade students, where 100 indicates the maximum possible score. Because the test scores used in this analysis are given only to sixth grade students, it is necessary to provide an outcome for secondary education. Following other studies (see Hanushek [16]), the drop out rate is included as an outcome of secondary education. (The inverse of the drop out rate was used in the programming models).

After selecting outcomes, it is necessary to specify both discretionary and environmental inputs. As regards discretionary inputs, the New York Depar tment of Education's Fiscal Profile and Basic Education System provides information to specify several measures of capital and labor inputs. The labor inputs chosen for this study include per pupil expenditures on teacher salaries, on pupil personnel instruction, and all other spending related to instruction. It is important to

Table 4 Descriptive statistics a (N = 556)

Variable Mean Standard deviation

Outcomes: Reading score 81.93 3.38 Math score 66.08 5.91 Social studies score 68.15 9.97 Drop out rate 2.62 1.91

Discretionary inputs: Teacher salary expenditures ($) 3150 666 Pupil personnel instructional expenditures ($) 173 87 All other instructional expenditures ($) 357 140 Books 22.77 8.06 Micro-computers 0.09 0.03

Environmental input: Adults with college education (%) 19.88 11.82

a Test score outcomes are measured as the average score on Pupil Evaluation Program tests given to all sixth grade students in 1990-91. All expenditures, measured in dollars per pupil, have been adjusted to control for cost of living differences. Books and micro-computers are measured per pupil. Source: New York State Department of Education, Compre- hensive Assessment Report and Basic Education Data Sys- tem; computations by author.

point out that expenditures capture not only inputs but also input prices. Thus, it is necessary to adjust expenditures to control for resource price differentials. Toward this end a teacher salary index was estimated. This index was estimated from a sample of individual teachers adjusting for differences in teacher experience, education, and certification to capture cost of living differences between school districts. Further, these compo- nents of instructional expenditures capture only labor inputs. Books and microcomputers, measured in per pupil units, were included as measures of capital.

To control for differences in the production environment, the percentage of adults graduating from college was included as an exogenous factor production. This variable, used as a proxy for parental education, has been found to consis- tently influence student performance. In an analysis of education production functions, Gyimah- Brempong and Gyapong [15] find that the education of the adult population is the only variable that can be used to represent all exogenous community characteristics that influence educational production. Descriptive statistics are reported in Table 4.

5.2. Empirical results

The modified VRS model was employed to estimate technical inefficiency in the provision of educational services. As discussed above, previ- ous empirical analyses have not properly con- trolled for environmental affects. This section discusses the results, which are reported in Table 5, that were obtained from the analysis.

Regarding technical inefficiency it was found that 443 districts (approximately 80% of the school districts) were inefficient. These inefficient districts could have provided the same level of services, on average, by using 80% of the observed level of inputs. In addition, half of these districts were only 71% efficient on average, sug- gesting a significant amount of technical inefficiency. An interesting but expected finding emerges regarding the relationship between the production environment and the level of ineffi-


Table 5 Mean efficiency results a (N = 556)

Classification Number Technical Adults with of districts efficiency college education

Cost efficiency class: Efficient districts 113 1.00 12.84

(0) (7.62) Inefficient districts 443 0.80 21.63

(0.12) (12.04) High inefficiency 221 0.71 24.61

(0.08) (12.76) Low inefficiency 222 0.89 18.68

(0.05) (10.50) All districts 556 0.84 19.85

(0.13) (11.82)

a Technical efficiency is estimated using the variable returns to scale modified model (7). Standard deviations are reported in parentheses. Source: Computations by author.

ciency. As shown in the table, efficient districts face a harsher environment on average than do inefficient districts. This highlights the important consideration that some districts may be identified as technically efficient relative to other districts by default due to environmental control.

6. Conc lus ions

This paper has provided a description of the technology for the public sector that is consistent with empirical analyses. This description provided the basis for developing a modified D E A model that will properly control for exogenous environmental variables that affect production. Further, this description allowed a theoretical critique of the existing D E A model that purportedly controls for exogenously fixed factors of production. It was proved that the BM model may lead to biased results due to the inclusion into the reference set of DMUs with a more favorable environment. Consequently, a modified linear programming model was developed consistent with the underlying technology that appropriately restricts the reference set. Using simulated data it was shown that the new D E A model outper- formed the BM model in measuring the level of technical efficiency. Unlike the BM model, the modified model was able to correctly identify all

efficient D M U s and achieve a high correlation with the know level of inefficiency. The results suggest that future research analyzing efficiency of public sector service provision should be consistent with known propert ies of the underlying technology. In addition, future research should address other implications that multiple frontiers has on nonparametr ic estimation. Also, it would be worthwhile to link nonparametr ic estimation of efficiency to organizational theories of inefficiency.

Acknowledgements

The author would like to acknowledge the help of Stuart Bretschneider, William Duncombe, Susan Gensemer, Mark Miller, Jerry Miner and John Yinger. In addition, I would like to thank two anonymous referees who provided valuable suggestions that improved this paper consider- ably.

References

[1] Ali, A.I., and Seiford, L.M., "The mathematical programming approach to efficiency analysis", in: H.O. Fried, C.A.K. Lovell and S.S. Schmidt (eds.), The Measurement of Productive Efficiency, Oxford University Press, New York, 1993, 120-159.


[2] Banker, R.D., Charnes, A., and Cooper, W.W., "Some models for estimating technical and scale inefficiencies in Data Envelopment Analysis", Management Science 30 (1984) 1078-1092.

[3] Banker, R.D., Gadh, V.M., and Gorr, W.L., "A Monte Carlo comparison of two production frontier estimation methods: Corrected ordinary least squares and data envelopment analysis", European Journal of Operational Research 67 (1993) 332-343.

[4] Banker, R.D., and Morey, R., "Efficiency analysis for exogenously fixed inputs and outputs", Operations Re- search 34 (1986) 513-521.

[5] Banker, R.D., and Morey, R., "The use of categorical variables in Data Envelopment Analysis", Management Science 32 (1986) 1613-1627.

[6] Banker, R.D., and Thrall, R.M., "Estimation of returns to scale using Data Envelopment Analysis", European Journal of Operational Research 62 (1992) 74-84.

[7] Bessent, A., and Bessent, E., "Determining the compara- tive efficiency of schools through Data Envelopment Analysis", Educational Administration Quarterly 16 (1980) 57-75.

[8] Bessent, A., Bessent, W., Kennington, J., and Reagan, B., "An application of mathematical programming to assess productivity in the Houston Independent School District", Management Science 28 (1982) 1335-1366.

[9] Bradford, D., Malt, R., and Oates, W., "The rising cost of local public services. Some evidences and reflections", National Tax Journal 22 (1969) 185-202.

[10] Charnes, A., Cooper, W.W., and Rhodes, E., "Measuring the efficiency of decision making units", European Jour- nal of Operational Research 2 (1978) 429-444.

[11] Debreu, G., "The coefficient of resource utilization", Econometrica 19 (1951) 273-292.

[12] Fiire, R., Grosskopf, S., and Lovell, C.A.K., The Measure- ment of Efficiency of Production, Kluwer-Nijhoff, Boston, MA, 1985.

[13] F~ire, R., Grosskopf, S., and Weber, W., "Measuring school district performance", Public Finance Quarterly 17 (1989) 409-428.

[14] Farrell, M.J., "The measurement of productive efficiency", Journal of the Royal Statistical Society. Series A. General 120 (1957) 253-281.

[15] Gyimah-Brempong, K., and Gyapong, A.O., "Character- istics of education production functions: An application of canonical regression analysis", Economics of Educa- tion Review 10 (1991) 7-17.

[16] Hanushek, E., "Conceptual and empirical issues in the estimation of educational production functions", The Journal of Human Resources 14 (1979) 351-388.

[17] Hanushek, E., "The economics of schooling: Production and efficiency in public schools", Journal of Economic Literature 24 (1986) 1141-1177.

[18] Jessoon D., Mayston, D., and Smith, P., "Performance assessment in the education sector. Educational and economic perspectives", Oxford Review of Education 13 (1987) 249-266.

[19] Kamakura, W.A., "A note on the use of categorical variables in Data Envelopment Analysis", Management Science 34 (1988) 1273-1276.

[20] Koopmans, T.C., "An analysis of production as an efficient combination of activities", in: T.C. Koopmans (ed.), Activity Analysis of Production and Allocation, Cowles Commission for Research in Economics, Monograph No. 13, Wiley, New York, 1951.

[21] Lovell, C.A.K., "Production frontiers and productive efficiency", in: H.O. Fried, C.A.K. Lovell and S.S. Schmidt (eds.), The Measurement of Productive Efficiency, Oxford University Press, New York, 1993, 3-67.

[22] Maindiratta, A., "Largest size-efficient scale and size efficiencies of decision-making units in Data Envelop- ment Analysis", Journal of Econometrics 46 (1990) 57-72.

[23] McCarty, T., and Yaisawarng, S., "Technical efficiency in New Jersey school districts", in: H,O. Fried, C.A.K. Lovell and S.S. Schmidt (eds.), The Measurement of Pro- ductive Efficiency, Oxford University Press, New York, 1993, 271-287.

[24] Seiford, L., and Thrall, R., "Recent developments in DEA, the mathematical programming approach to Fron- tier Analysis", Journal of Econometrics 46 (1990) 7-38.

Documents

On the measurement of technical efficiency in the public sector